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Abstract 
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In tiiis paper, we investigate the mean field games with K classes 
of agents who are weakly coupled via the empirical measure. The 
underlying dynamics of the representative agents is assumed to be a 
controlled nonlinear Markov process associated with rather general 
integro-differential generators of Levy-Khintchine type (with variable 
coefficients), with the major stress on applications to stable and stable- 
like processes, as well as their various modifications like tempered 
stable-like processes or their mixtures with diffusions. We show that 
nonlinear measure-valued kinetic equations describing the dynamic 
law of large numbers limit for system with large number of agents 
are solvable and that their solutions represent 1 /A^-Nash equilibria for 
approximating systems of N agents. 
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1 Introduction 



1.1 Main objectives 

The mean-field game (MFG) methodology represents one of the gems in the 
recent progress of stochastic control. It was developed independently by J.- 
M. Lasry and P.-L. Lions in France (where the term 'mean- field game' was 

^Supported by the AFOSR grant FA9550-09-1-0664 'Nonlinear Markov control pro- 
cesses and games' 
2arXiv:1112.3744v 



1 



coined, see [50], [SI], [S2],[33]) and by M. Huang, R.P. Malhame and P. Caines 
in Canada, where it was initially called the 'Nash certainty equivalence princi- 
ple', see [35], [36], [37], [38], [39], [IQ]. Mean-field game methodology aims at 
describing control processes with large number N of participants by studying 
the limit N ^ oo when the contribution of each member becomes negligi- 
ble and their interaction is performed via certain mean-field characteristics, 
which can be expressed in terms of empirical measures. A characteristic fea- 
ture of the MFG analysis is the study of a coupled system of a backward 
equation on functions (Hamilton- Jacobi-Bellman equation) and a forward 
equation on probability laws (Kolmogorov equation). The work on mean- 
field games so far was performed mostly for the underlying Markov process 
(describing individual evolutions) being a simple diffusion, where the method 
of McKean-Vlasov SDEs (describing the so called nonlinear diffusions) was 
available. 

Meanwhile one of the authors of the present paper suggested in [51] the 
program of studying nonlinear Markov processes that describe the dynamic 
laws of large numbers for general Markov systems of interacting particles 
extending and unifying in a natural way various models of natural science 
including McKean-Vlasov diffusions, Smoluchovski and Boltzman evolutions, 
replicator dynamics of evolutionary games and many others. This program 
was then developed in some detail in monograph [52]. In this paper we 
aim to show that the machinery of nonlinear Markov processes can serve 
as a natural tool for studying mean-field games with the general underly- 
ing Markov dynamics of agents (not only diffusions). More specifically, the 
main consistency equation of MFG can be looked at as a coupling of a non- 
linear Markov process with certain controlled dynamics. Using this link we 
develop the MFG methodology for a wide class of underlying Markov dynam- 
ics including in particular stable and stable-like processes, as well as various 
their modifications like tempered stable-like process or their mixtures with 
diffusions. 

Moreover, our abstract approach yields essential improvements even for 
underlying processes being diffusions. In particular, as compared with [57] . 
it includes the case of diffusions coefficients (not only drifts) depending on 
empirical measures, it allows us to get rid of the assumption of small coupling 
(or composite gain), to prove (rather than just assume) the crucial sensitivity 
estimates (feedback regularity condition (37) in [37]), and finally to get a full 
prove of convergence rate of order 
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1.2 The basic setting of MFG methodology and the 
strategy for its implementation 

Let us explain now the main ideas, objectives and strategy of our analysis. 
Suppose a position of an agent is described by a point in a locally compact 
separable metric space X. A position of agents is then given by a point in 
the power X'^ = X x ■ ■ ■ x X {N times). Hence the natural state space for 
describing the variable (but not vanishing) number of players is the union 
X — U^^X^. We denote by Cgymi'^^) the Banach spaces of symmetric (with 
respect to permutation of all arguments) bounded continuous functions on 
X^ and by Csym('^) the corresponding space of functions on the full space 
X. We denote the elements of X by bold letters, say x, y. 

Reducing the set of observablcs to Csym('^) means effectively that our 
state space is not X (or X^ in case of a fixed number of particles) but rather 
the quotient space -S"^ (or SX^ resp.) obtained with respect to the action 
of the group of permutations, which allows the identifications Csyui{X) — 
C{SX) and Csym(^^) = C{SX^). Clearly SX can be identified with the 
set of all finite collections of points from X, the order being irrelevant. 

A key role in the theory of measure- valued limits of interacting particle 
systems is played by the inclusion SX to V{X) (the set of probability laws 
on X) given by 

1 1 

X = (xi, Xn) ^ -^((^xi H ^^xn)^ ;^<^x, (1-1) 

which defines a bijcction between SX'^ and the subset V^{X) (of normalized 
sums of Dirac's masses) of V{X). This bijection extends to the bijection of 
SX to 

Vs{X) := U^^,V!'{X)GV{X), 

that can be used to equip SX with the structure of a metric space by pulling 
back any distance on V{X) that is compatible with its weak topology. 

Remcirk 1.1. With a slight abuse of notation, we use 5x to denote the sum 
of the Dirac's measures on X, i.e. 

:= H h Sj:^. 

So 5x is not the Dirac measure on X with support at point x. 
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Let {y4[)f:, /i, m]} be a family of generators of Feller processes in X, where 
t > 0, /i G V{X) and u eU {a metric space interpreted as a set of admissible 
controls). Assume also that a mapping 7 : R"*" x X ^ U is given. For 
any N, let us define the following (time-dependent) family of operators (pre- 
generators) on Csymi'^^) describing N mean-field interacting agents: 



N 



i=l 

N 



lf[7]/(x) = lf[7]/(xi,-- - ,x;v) :=J2A%f,,u,]f{x^r-- ,xn), (1-2) 
where 

i=l 

is the empirical distribution of agents, = 7(t,Xi) and A''[t, fi,Ui]f means 
the action of the operator A[t,fi,Ui] on the ith variable of the function /. 
Let us assume that the family ^^[7] generates a Markov process = 
{X^{t) = (Xf (t), . . . ,Xj^(t) : t > 0)} on for any N. We shall refer to 
it as a controlled ( via control 7 ) process of N mean-field interacting agents. 

Remark 1.2. In the terminology of statistical mechanics the operator At['j] 
(considered for all N, i.e. lifted naturally to the whole space Cgymi'^) ) should 
he called the second quantization 0/ /x, m] . 

Using mapping fll.lj) . we can transfer our process of N mean- field inter- 
acting agents from SX^ to Vf {X). This leads to the following operator on 
C{Vf{X)): 

N 

A^Yl]F{5^/N) = If [7]/(x) = Y,A%ft,u.]f{x,, ■ ■ ■ ,xn), (1-3) 

i=l 

where /(x) = F{6^/N) and x = (xi,-- - ,xn). Let us calculate the action of 
this operator on linear functionals F, that is on the functionals of the form 

F^{fi) = {g,fi) = I g{x)fi{dx) (1.4) 
for a (jf G C{X). Denoting 5'®(x) = X^ili di^i) for x = (xi, ■ ■ ■ Xjy) we get 
Af[7]F^(5x/A^) = ^ (If [7]^7®) (a:i,--- ,x^) 

1 A (^-^^ 
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Hence, if /j,^ — 5j^/N — > //^ e V{X) as — > oo, we have 

A^^[^]F^{SjN)^{A[t,i,^,^{t,.)]g,i,^), as N ^ oo, 
so that the evolution equation 

Ft = If [7]i^t (1-6) 

of our controlled process of mean-field interacting agents, for the linear 
functionals of the form — {g,iit{l^)) turns to the equation 



[1.7) 



We call this equation the general kinetic equation in weak form. It should 
hold for g from a suitable class of test functions. This limiting procedure will 

be discussed in detail later on. 

Let us explain how the mapping 7 pops in from individual controls. As- 
sume that the objective of each agent is to maximize (over a suitable class 
of controls {u.}) the payoff 



E 



Jis, (5), /.f , Us) ds + V^{X^{T)) 



consisting of running and final components, where the functions J : R"*" x 
X X V{X) X W — )■ R and : A" — ?> R, and the final time T are given, and 
where {11.} is the family of the empirical measures of the whole process 



N 



1 



is) 
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), t<s<T. 



By dynamic programming (and assuming appropriate regularity), if the dy- 
namics of empirical measures /ig is given, the optimal payoff 



V^lt, x) = supE 



jT^ J{s, X(s), /xf , u,) ds + V^{X{T)) 



of an agent starting at x at time t should satisfy the HJB equation 
dVMit,x) 



dt 



+ max (J(t, X, /xf , u) + A[t, /if, M]1/jv(t, x)) = (1.8) 
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with the terminal condition V]\f(T,.) = V^i^-). If fit & V{X) as 

N ^ CO, then it is reasonable to expect that the solution of fll.Sp converges 
to the solution of the equation 

dV(t x) 

— + max ( J(t, X, /it, u) + A[t, fXt,u]V{t, x)) = 0. (1.9) 

Remark 1.3. To shorten the formulas used below for the analysis of HJB 
equation fll.Sp . we assume that the final cost function depends only on 
the terminal position X{T), but not on fix- Only minor modifications are 
required, if does depend on fiT- 

Assume HJB equation ( ]1.9p is well posed and the max is achieved at one 
point only. Let us denote this point of maximum by u = T{t,x, {fi>t})- 

Thus, if each agent chooses the control via HJB (11. 9p . given an empirical 
measure ft, i.e. with 

-f{t,x) = Tit,x,{fi>t}), (1.10) 

this 7 specifies a nonlinear Markov evolution {fit}t>o via kinetic equation 
(11. 7p . The corresponding MFG consistency (or fixed point) condition {/i.} = 
{yU.} leads to the equation 

^(^>/^t) = (^[^,/^i,r(t, .,{/i>t})]^,/it), (1.11) 

which expresses the coupling of the nonlinear Markov process specified by 
(II. lip and the optimal control problem specified by HJB (II. 9p . It is now 
reasonable to expect that if the number of agents N tends to infinity in such 
a way that the limiting evolution is well defined and satisfies the limiting 
equation (II. IIP with F chosen via the solution of the above HJB equation, 
then the control 7 and the corresponding payoffs represent the e-Nash equi- 
librium for the controlled system of N agents, with e — )■ 0, as — > 00. This 
statement (or conjecture) represents the essence of the MFG methodology. 

Under certain assumptions on the family A\t, /i, m], we are going to justify 
this claim by carrying out the following tasks: 

Tl) Proving the existence of solutions to the Cauchy problem for coupled 
kinetic equations (II. lip within an appropriate class of feedback F and the 
well-posedness for the uncoupled equations (II. 7p . Notice that we are not 
claiming uniqueness for (11.110 . It is difficult to expect this, as in general Nash 
equilibria are not unique. At the same time, it seems to be an important 
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open problem to better understand this non-uniqueness by describing and 
characterizing specific classes of solutions. On the other hand, well-posedness 
for the uncoupled equations (11.71) is crucial for further analysis. 

T2) Proving the well-posedness of the Cauchy problem for the (backward) 
HJB equation (II. 9p . for an arbitrary fiow {/i.} in some class of regularity, 
yielding the feedback function F in the class required by Tl). This should 
include some sensitivity analysis of F with respect to the functional parameter 
{/i.}, which will be needed to show that approximating the limiting MFG 
distribution {//.} by approximate A^-particle empirical measures yields also 
an approximate optimal control. To perform this task, we shall assume here 
additionally that the operators A[t,fi,u] in (II. 9p can be decomposed into 
the sum of a controlled 1st order term and a term that does not depend on 
control and generates a propagator with certain smoothing properties. This 
simplifying assumption allows to work out the theory with classical (or at 
least mild) solutions of HJB equations. Without this assumption, one would 
have to face additional technical complications related to viscosity solutions. 

T3) Showing the convergence of the A^-particle approximations, given by 
generators (II. 5p to the limiting evolution (II. 7p . i.e. the dynamic laws of large 
numbers (LLN), for a class of controls 7 arising from (ll.lOp with a fixed {fi-}, 
where F is from the class required for the validity of Tl) and T2). Here one 
can use either more probabilistic compactness and tightness (on Skorokhod 
paths spaces) approach, or a more analytic method (suggested in [SI]) via 
semigroups of linear operators on continuous functionals of measures. In this 
paper, we shall use the second method, as it yields more precise convergence 
rates. For the analysis of the convergence of the corresponding semigroups 
the crucial ingredient is the analysis of smoothness (sensitivity) of the so- 
lutions to kinetic equations (ll.7p with respect to initial data. The rates of 
convergence in LLN imply directly the corresponding rather precise estimates 
for the so-called propagation of chaos property of interacting particles. 

T4) Finally, combining T2) and T3), one has to show that thus obtained 
strategic profile (ll.lOp with {fi.} = {fi.} represents an e- equilibrium for 
agents system with e — )• 0, as — > 00. Actually we going to prove this with 
e = 1/N using the method of tagged particles in our control setting. 

Remark 1.4. A similar, but slightly different statement of the MFG method- 
ology would be the claim that a sequence of Nash equilibria for N particle 
game converges to a profile F solving (II. lip . By our methods it would not be 
difficult to justify this claim if existence and uniqueness of a Nash equilibrium 
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for each N -player stochastic differential game were given. Our formulation 
of MFG methodology allows us to avoid discussion of this nontrivial question. 

1.3 Discrete classes of agents 

Let us specify our model a bit further. 

Of particular interest are the models with the one-particle space X having 
a spatial and a discrete components, the latter interpreted as a type of an 
agent. Thus let X = R'^x /C, where /C is either a finite or denumerable set. In 
this case, functions from C{X) can be represented by sequences / = {fi)i£K 
with each G C(R'^), the probability laws on X are similarly given by the 
sequences /i = of positive measures on R'' with the masses totting up 

to one. 

The operators A in C{X) are specified by operator- valued matrices {Ajj}, 
2,j G /C, with Aij being an operator in C(R'^), so that {Af)i = Xljg/c^u/?- 
It is not difficult to show (see [11] for details) that for such a matrix A to 
define a conditionally positive conservative operator in C{X) (in particular, 
a generator of a Feller process) it is necessary that Aij for i ^ j are integral 
operators 

{A,f){z)= [ {f){y)-f{z))u,,{z,dy) 

with a bounded (for each z) measure Vij{z,dy)., and the diagonal terms are 
given by the Levy-Khintchin type operators {i E K,): 

Aufiz) = ^-{G,{z)V,V)f{z) + {b,{z),Vf{z)) 

+ [ {fiz + y)-f{z)-iVf{z),y)lBM)Mz,dy), (1.12) 

with Gi{z) being a symmetric non-negative matrix, Uilz,.) being a Levy 
measure on R'^, i.e. 

/ mm{l, \y\^)ui{z, dy) <oo, zy({0}) = 0, (1.13) 

depending measurably on z, and where Ib^ denotes, as usual, the indicator 
function of the unit ball in R'^. 

Operators Aij with i j describe the mutation (migration) between 
the types. Leaving the discussion of mutations of infinite number types to 
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another publication, we shall concentrate here, for simplicity, on the case 
when /C is a finite set {1, ■ ■ ■ , K} and no mutations are allowed, that is all 
non-diagonal operators Aij with i ^ j vanish. Hence A will be given by a 
diagonal matrix with the diagonal terms Ai = An of type fll.l2p . 

Let us assume additionally that each agent can control only its drift, that 
is the diagonal generators have the form 

Ai[t,fi,u]f{z) = {hi{t,z,fi,u),Vf{z)) + Li[t,fi]f{z), t = l,--- ,K, (1.14) 

with Li of form (11.121) . i.e. 



LS, fi]f{z) = ^{G,{t, z, fi)V, V)f{z) + {b,{t, z, /i), Vf{z)) 

+ [ UX^ + y) - fiz) - i^f{z),ynBMMt,z,fi,dy) 



:i.i5) 



with the coefficients Gi, bi, z/j depending on t G R"*" and /i = (/^i, ■ ■ ■ , /ix) G 
V{X) as parameters. 

Remark 1.5. //, for a given (probability) measure flow {f^t}te[o,T]! the opera- 
tors L[t, fit] = (Li, ■ ■ ■ , LK)[t, fit] generate a Markov process {Rt[fit]}te[o,T] = 
{{Rjlfit], ■ ■ ■ , Rf[fit])}te[o,T], one can write a stochastic differential equation 
(SDE) corresponding to the generator given in (TA^ as 

dXl = hi(t, XI, fit, ul) dt + dRllfit], i = 1, - ■ ■ ,K. 

If fit are required to coincide with the laws of XI, for all t G [0,T], these 
equations take the form of SDEs driven by nonlinear Levy noises, developed 
in f5^ and Chapter 3 of f5^ . 

The initial work on the mean field games, done by Lions et al. and Gaines 
et ai, dealt with the processes Rtlfj] being Brownian Motions without depen- 
dence on fi. In our framework, this underlying process is extended to an 
arbitrary Markov process with a generator U.15\) depending on fi. 

In the main kinetic equation (11.71) . we shall then have (g, fit) = Y^f=i{9i-, Ii'i,t) 
and 

A[t, fit, -fit, .)]g = {Ai[t, fit, -fit, .)]gi}f=i 

with 

A[t,fi,lit,.)]giiz) = {hiit, z, fi,-f{t, .)),Vgiiz)) + Li[t, fi]giiz). (1.16) 
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HJB equation f 1 1.9 1) now decomposes into a collection of HJB equations 
for each class of agents, written as 

dVUt x) 

^ + Hl{x,VV'{x),i2t) + LS,fit]V'{t,x) = (1.17) 

where 

Hl{x,p,^t) ■.= max{hi{t,x,fit,u)p+Ji{t,x,fit,u)}. (1-18) 

We have assumed the resulting feedback control is unique (i.e. argmax in 
fll.lSp is unique). Let us give two (related) basic examples of such a situation. 

Example 1.1 {HoQ-optima\ control, see [70] for its systematic presentation). 
For each i, the running cost function Ji is quadratic in u, i.e. 

Ji(t, X, fi, u) = ai(t, X, /i) — 9i(t, X, fi)u^ 

and the drift coefficient hi is linear in u, i.e. 

hi{t,x,iJ,,u) = Pi{t,x,fi)u, 

where the functions ai,Pi,di : [0,T] x R'^ x V(R'^) R and 6i{t,x,fi) > 
for any {t,x,^). Thus we are maximising a quadratic function over control 
u. It is easy to get an explicit formula of the unique point of maximum, i.e. 

u = ^{t,x,^l)p. 



Thus HJB equation ( [i.i7| ) rewrites as 



' + ^(t, X, fx){Wf{t, x) + «,(t, X, /i) + U[t, iit]V\t, x) = 
which is a generalized backward Burger's equation. 

Example 1.2. Assume, for each i, hi{t,x, n,u) = u and Ji{t,x, n,u) is a 
strictly concave smooth function of u. Then HI is the Legendre transform 
of —J as a function of u, and the unique point of maximum in (11.181) is 
therefore u = dHl/dp. Then kinetic equation (11. lip takes the form 

d dH^ 

'jSd^^t) = ^{Li{t,^t)gi + -Q^{x,p, ^t)\p=vv-{x)^ gr, ^i,t)- (1.19) 
fc=l ^ 
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If K = 1 and Ji{t,x, fi,u) has the decomposition 

Ji(t, X, /i, u) = V{x, fi) + J{x, u) 

for V : IV^ X V(R'^) and J : R'* x U, and Li{t,fi) = A, the corresponding 
system of coupled equations fll.lQp and fll.l7p turns to system (2) of fU^ 
( only there the kinetic equation is written in the strong form and in reverse 
time). 

Concrete examples, for which our abstract results hold, are presented in 
Section HI 

1.4 Plan of the paper 

The rest of the paper is organized as follows. In Section we prove basic 
existence and uniqueness results for the extension of forward kinetic equation 
(11.71) obtained by coupling 7 with {/it}te[o,r]- In Section [3l we analyze more 
specifically equation (11. 7p without any coupling and give a probabilistic in- 
terpretation of its solutions as nonlinear Markov processes, as well as some 
related regularity results. Then the basic examples are discussed in Section 
m In Section [5|, we are concerned with the well-posedness of the backward 
HJB equation (11.90 . i.e. the existence and uniqueness of mild solutions to the 
equation with the given terminal boundary condition. Our main additional 
assumption will be a smoothing property of the semigroup of an underlying 
Markov process that defines an uncontrolled part of the evolution of agents 
(generated by the operators Li in (I1.14p ). Sensitivity analysis on the solution 
of this HJB equation will be carried out in Section [6] completing in essence 
tasks Tl) and T2) outlined above in Subsection 11.21 for models with discrete 
classes from Subsection II. 3[ Section [7] deals with smooth dependence (sen- 
sitivity) of nonlinear Markov processes with respect to initial data. This is 
crucial for the proof of the dynamic laws of large numbers (LLN) established, 
in an appropriate form with rates of convergence, in Section [8] solving task 
T3) from Subsection II. 2[ Section [9] completes our programme by justifying 
the approximation of players stochastic games by a mean- field limit. 

1.5 Main notations 

The following basic notations will be used: 
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For a Banach space B, B* is the dual Banach space with the norm 

||/i||B- = SUp,|^,|g<i 

For a Unear operator L between the Banach spaces D and B, we use the 
standard notation for its norm 

II-^'IIdm-b = sup \\Lg\\-B] 

llffl|D=l 

in particular, ||i>||B = ||-^||bm.b 

C([0, T], B) is the Banach space of continuous functions t — )■ /Xt e B with 

the norm sup^gjo^^] || /it Hb 

If is a closed subset of B, then 

C([0, T], A^(B)) is a closed subset of C([0, T], B), consisting of continuous 
functions t ^ lit & A^(B); using A^(B) rather than simply JH, stresses the 
topology used 

CLip{[0, T],M{B)) is a subset of C([0, T],M{B)), consisting of Lipschitz 
continuous functions t ^ fit E A^(B) 

C^([0,T],B) (resp. C^([0, T], yW(B))), a closed subset of C([0,T],B) 
(resp. C([0, T], A^(B))), consists of continuous curves G B (resp. ^ 
A^(B)), s e [0, T] with a fixed initial data Co = 1^ & M 

Cj([0,T],B) (resp. Cj([0, T], M(B))), a closed subset of C([0,T],B) 
(resp. C([0, T], A^(B))), consists of continuous curves G B (resp. £ 
A^(B)), s G [0,T] with a fixed terminal data = A* 

For a locally compact metric space X, 

Cao{X) is the Banach space of bounded continuous functions / on A" with 
lim-r^oo f{x) = 0, equipped with sup-norm 

M^*^"(A'), the dual space of Coo (A), is the Banach space of signed Borel 
measures on X; {f,fi) = f f{x)fj,{dx) denotes the usual pairing 

M+(A'), a subset of M'**^'*(A'), is the set of positive Borel measures on X 

V{X) is the set of probability measures on X 

CLjp(R'^) is the Banach space of bounded Lipschitz continuous functions 
/ on with the norm \\f\\u, = sup, \f{x)\ + sup,^, ^^^t-T 

C"(R'^) is the Banach space of n times continuously different iable and 
bounded functions / on R'^ such that each derivative up to and including 
order n is bounded, equipped with norm ||/||c" which is the supremum of 
the sums of all the derivatives up to and including order n 

C^(R'') is a closed subspace of C"(R'') with / and all its derivatives 
up to and including order n belonging to C^(R,^)] sometimes we use short 
notations, and C^o, respectively 
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2 Abstract anticipating kinetic equations 



This section extends some existence and uniqueness results for nonlinear 
Markov processes from [52] to a rather general anticipating (non-Markovian) 
setting. 

Let D be a dense subset of B, which is itself a Banach space with the 
norm || ||d > || ||b- 

A deterministic dynamic in the dual Banach space B* can be naturally 
specified by vector-valued ordinary differential equation 

flt = Q{t,fit) (2.1) 

with given initial value /io G B*, where is a nonlinear (possibly unbounded) 
operator in B*. One can write equation (12. ip in the weak form as the equation 

(/,/i,) = (/,^^(t,/it)) (2.2) 

that must hold for some class of test functions / G B. In many applications, 
equation ( 12. 2 p can be represented in the form 

^(/,/it) = (A[t,^i]/,^i) (2.3) 

where the mapping {t,T]) i— ^[t,'/]] is from R"*" x B* to linear operators 
y4[)f:,?7] : D I— )• B and for each pair (t,ri), A[t,ri] generates a semigroup in 
B. We call equation (12. 3 p the general Markovian kinetic equation, as it 
contains most of the basic equations of non-equilibrium statistical mechanics 
and evolutionary biology. For an extensive discussion of its properties and 
applications we refer to monograph Of major interest is the case when 
B* is the space of measures on a locally compact space X, i.e. B* = M''*^"(A') 
and ^[t, "Z]] generate Feller processes, but in this section we shall work in an 
abstract setting. 

For the study of MFG, we need a anticipating extension of equation (12. 3p . 
that is an anticipating or non-Markovian kinetic equation of the form 

j^{f,fit) = {A[t,{fis}o<s<T]f,fit) (2.4) 

where the mapping [t, {r].}) t— )■ A[t, {r].}] is from R+ x C^([0, T], B*) to linear 
operators A[t, {rj.}] : D B. 
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Two particular cases are of major importance. When the operators A only 
depend on the past, i.e. {fis}o<s<T = {/^<t}; equation f l2.4p can be called a 
non- anticipating kinetic equation; when the generators A only depend on the 
future, i.e. {/is}o<s<r = {fJ^>t}, equation (12. 4 p can be called a forecasting 
kinetic equation. 

Non-anticipating equations can be seen as analytic analogs of SDE with 
adapted coefficients, and their well-posedness can be obtained by similar 
methods. But for MFG, we need much more complicated forecasting equa- 
tions. Here we prove local well-posedness and global existence results for 
general equations (12. 4 p (including forecasting ones) and global well-posedness 
for non- anticipating case. 

Let us start by recalling the basic properties of propagators. For a set 
S, a family of mappings U^'^ from S to itself, parametrized by the pairs of 
numbers r <t (resp. t < r) from a given finite or infinite interval is called a 
(forward) propagator (resp. a backward propagator) in S, if is the identity 
operator in 5* for all t and the following chain rule, or propagator equation, 
holds for r < s <t (resp. for t < s < r): 

jjt,sjjs,r _ jjt,r 

A backward propagator f/*'*" of bounded linear operators on a Banach 
space B is called strongly continuous if the operators depend strongly 
continuously on t and r. By the principle of uniform boundedness if U^'^ is a 
strongly continuous propagator of bounded linear operators, then the norms 
of t/*'^ are bounded uniformly for t, r from any compact interval. 

Suppose U^'^ is a strongly continuous backward propagator of bounded 
linear operators on a Banach space with a common invariant domain D. Let 
Lt, t > 0, be a family of bounded linear operators D i— )■ B depending strongly 
measurably on t. Let us say that the family Lt generates t/*'^ on D if, for 
any / G D, the equations 

Af/*.7 = U^'^LJ, ^U^'^-f = -L,f/^'7, 0<t<s<r, (2.5) 
as as 

hold a.s. in s (with the derivatives taken in the topology of B), that is there 
exists a set of zero-measure in R such that for all t < r and all / G D 
equations (12. 5p hold for all s outside S, where the derivatives exist in the 
Banach topology of B. In particular, if the operators Lt depend strongly 
continuously on t (which will be always the case in this paper), equations 



14 



fl2.5p hold for all s and / G D, where for s = t (resp. s = r) it is assumed to 
be only a right (resp. left) derivative. In the case of propagators in the space 
of measures, the second equation in (12.51) is called the backward Kolomogorov 
equation. 

One often needs to estimate the difference of two propagators when the 
difference of their generators is available. To this end, we shall often use the 
following rather standard trick that we formulate now in abstract form. 

Proposition 2.1. Let LI, i = 1,2, t > 0, be two families, continuous in t, of 
bounded linear operators D i— t- B (D and B are two Banach spaces equipped 
with a continuous inclusion D — t- B^ and Ul'"^ be two backward propagators 
in B with \\UI'^\\-b < Ci, i = 1,2, s.t. for any / G D equations (12. 5p hold in 
B for both pairs {Li,Ui). Let D be invariant under Ul'^ and \\UI''^\\d < C2- 
Then 

U'/ - U'/ = U',\Ll - LlW/ds (2.6) 

and 

11^2''' - ^i'''I|d-->b < ciC2(r - t) sup - L^IId^b- (2.7) 

t<s<r 

Proof. Define an operator-valued function Y[s) := U2^Ui'^ . Since are 
identity operators, Y{r) = U2'' and Y{t) = U^'^ . By (12. 5p . we get 

= 1^ u'/Liur - u'2'Liurds 

U'2\Ll-L\)Ul'^ds, 

which implies both (jM]) and ([22]). □ 

As a consequence we get a simple result on the continuous dependence 
of propagators on parameters that can be considered as a starting point for 
later developed sensitivity. 

Proposition 2.2. Let {L\t,a\ : t > 0, a G R} be a family, continuous in 
t and a, of bounded linear operators D i-)- B (D and B are two Banach 
spaces equipped with a continuous inclusion D — )• B^ and U^f be backward 
propagators in B generated by on a common invariant domain D 

s.t. ||f/*'''||B and ||f/*'''||D are both uniformly hounded. Then the family {U^f} 
depends strongly continuously on a, as a family of operators in B. 
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Proof. From f l2.7p we conclude that U^^f is a continuous function a ^ B 
for any / G D. By the density of D in B (and boundedness of ||?7*'^||b) this 
property extends to all / G B. □ 

We are ready to present our results on the existence of solutions to general 
kinetic equations. 

Theorem 2.1 (local well-posedness for general anticipating case). Let Ai be 

a convex and bounded subset o/B* with snp^^j^ II/^IIb* < M, which is closed 
in the norm topologies of both B* and D*. Suppose that 

(i) the linear operators {^.}] : D t— )■ B are uniformly bounded and 
Lipschitz m {i.}, I.e. for any {i],{r].} G C^{\Q,T], MiT>*)) 

sup ||A[t,{e.}]-A[t,{77.}]||D^B <ci sup 116 -^^IId-, (2.8) 
te[o,T] ie[o,T] 

sup P[t,{e}]||D^B<ci (2.9) 
te[o,r] 

for a positive constant C\; 

(a) for any {6} G C^([0, T], A^(D*)), let the operator curve A[t, {6}] : 
D I— )■ B generate a strongly continuous backward propagator of bounded linear 
operators f/*''^[{6}] in'Q, ^ < t < s, on the common invariant domain D, 
such that 

||f/*'1{e}]||D^D < C2 and ||f/*'1{e.}]||B^B < C3, t< s, (2.10) 

for some positive constants €2,03, and with their dual propagators f/*'*[{6}] 
preserving the set Ai. 
Then, if 

C1C2C3MT < 1, (2.11) 
the weak nonlinear non-Markovian Cauchy problem 

^(/,/xJ = (A[t,{/i,},e[o,T]]/,/it), /io = /i, te[0,T], (2.12) 

is well posed, that is for any ^ E M., it has a unique solution $*(yu) G M. 
(that is l{2.1^) holds for all f E D) that depends Lipschitz continuously on 
time t and the initial data in the norm ofD*, i.e. 

||<l>^(/i) -<l>*(/i)||D. < CiC2(s-t), 0<t<s<T, (2.13) 
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and 



||{$ (/i)}-{$-(7^)}||c([0,T],D.)= sup ||<|.^(^)- $^(7^)110* 



se[o,T] 



(2.14) 



Proof. By duality 



By Proposition 12.11 with = A[t,{^\}], together with assumptions (12. 8p 
and fl2:T0|) . 



Consequently, if (12. lip holds, the mapping } 1— )■ {t^*'°[{C}]}te[o,T] is a 
contraction in C^([0, T], A^(D*)). Hence by the contraction principle there 
exists a unique fixed point for this mapping and hence a unique solution to 
equation (I2.12p . 

Inequality (I2.13P follows directly from (I2.12p . Finally, if $*(/x) = fit and 
$*(?]) = Tjt, then 



= (t/*'°[{/..}] - ^*'°[{r/.}])/x + ^*'°[{r].}](/x - v)- 
From fl2:T0|) and fl2J[5|l . 

- {^.}||C([0,T1,D*) < CiC2C3TM||{/i.} - {r/.}||c([0,Tl,D*) + C2\\fi - r/Ho* 

(2.16) 

implying (I2.14p . □ 



Remark 2.1. Condition (12. lip is the analog of the condition of small cou- 
pling (or composite gain) from [37]. 

As a consequence, we get global well-posedness for the Markovian case: 



\\{u'''[{e}]-u'''[{e}M\\B^ 




(2.15) 



<CiC2C3tM||{e.^}-{e.'}||c;([0,T],D*). 
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Theorem 2.2. Under the assumptions in Theorem \2.1[ hut without the lo- 
cality constraint fl2.11l) . the Cauchy problem for kinetic equation 

^(/, ^^t) = {A[t, fit]f, /it), /is = /i (2.17) 

is well-posed, i.e. for any fi G Ai, s G [0,T], it has a unique solution 
t/*'*(/i) G Ai, t G [s,T], and the transformations f/*'** of Ai form a propagator 
depending Lipschitz continuously on time t and the initial data in the norm 
ofD*, i.e. 

l|f/*'^(/.)-f/*'^(r^)||D. <c(T)||^-r/||D., (2.18) 
with a constant c{T) . 

Proof. Global solution is constructed by extending local solutions via itera- 
tions, as is routinely performed in the theory of ODE. □ 

It is not difficult to extend this result to non-anticipating equations yield- 
ing the following. 

Theorem 2.3. Under the assumptions in Theorem \2.1\ hut without the lo- 
cality constraint (12.111) . the non- anticipating nonlinear Cauchy problem 

^(/,/Xi) = {A[t, {^^<t]]f,^lt), /io = /i, t > 0, (2.19) 

is well posed in M. and its unique solution depends Lipschitz continuously on 
initial data in the norm ofY)*. 

Proof. For a /i G A^, let us construct an approximating sequence {^"} G 
C([0,T],A^(D*)), n = 0,1,---, by defining ^° = /i for t G [0,T] and then 
recursively 

er = f>*'°[e<7V, VtG[0,T]. 

By non-anticipation, arguing as in the proof of (12.1 5p above, we first get the 
estimate 

sup lie' -e? I Id- < ciCaCgMt, 

0<r<t 

and then recursively 

sup lie - c"'iid. < C1C2C3M / sup iier' - c"'iid. ds 

0<r<t Jo 0<r<s 
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that implies (by straightforward induction) that, for all t G [0,T], 

ll{e}-{C-'}llc([0,t],D*)= sup lle^^-er^llD. <^(ClC2C3Mt)^ 

0<r<t n\ 

Hence, the partial sums on the r.h.s. of the obvious equation 

c<t = ic<t - c<i') + ■■■+ {e<t - e<t) + e<t 

converge, and thus the sequence ^.'^ converges in C([0,T],D*). The limit is 
clearly a solution to (I2.19p . 

To prove uniqueness and continuous dependence on the initial condition, 
let us assume that fit and rji are some solutions with the initial conditions n 
and rj respectively. Instead of fl2.16p . we now get 

\\{f^.} - {v}\\c{[o,t],D*) < C1C2C3M / ll/i. - i],\\c{[o,s],D*) ds + C2\\fl -r]\\ii*- 

Jo 

By Gronwall's lemma, this implies 

\\M - {^.}||c([o,t],D*) < - v\\i^'e^"''''''\ t e [0,T] 

yielding uniqueness and Lipshitz continuity of solutions with respect to initial 
data. □ 
For the general anticipating kinetic equations, we have only the existence 
result. 

Theorem 2.4 (global existence of the solution for general anticipating case). 
Under the assumptions in Theorem \2.1\ hut without the locality constraint 
(12. lip , assume additionally that for any t from a dense subset of [0,T], the 
set 

: {i.}eC,{[Q,TlMm)} (2.20) 

is relatively compact in Ai. 

Then a solution to the nonlinear Cauchy problem 

^{f,f^t) = {A[t,{fIs}s&[o,T]]f,^^t), /^o = /i,i>0, (2.21) 

exists in Ai . 
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Proof. Since Ai is convex, the space C^([0, T], A^(D*)) is also convex. Since 
the dual operators preserve the set M, for any {^.} G C^([0, T], A^(D*)), 

the curve belongs to C^([0, T], A^(D*)) as a function of t. Hence, 

the mapping {^.} ^ {U''^[{C}]fi,t e [0,T]} is from C^([0, T], M(D*)) to 
itself. Moreover, by (12.151) . this mapping is Lipschitz continuous. 

Denote C = {{U-'^[{^.}]l^} : {^.} G C^([0, T], A1(D*))}. Together with 
f|2.13p . the assumption that set fl2.20p is compact in Ai for any t from a dense 
subset of [0, T] implies that the set C is compact (Arzela-Ascoli Theorem, 
see A.21 in [16]). 

Finally, by Schauder fixed point theorem, there exists a fixed point in 
C C C^([0, T],M(D*)), which gives the existence of a solution to ( H^ . □ 

In case of Markov processes, the compactness assumption (12.201) turns 
out to hold under very general conditions, see below Proposition 13.31 that 
covers all our examples. 



3 Nonlinear Markov processes in 

This section is designed to provide a probabilistic interpretation for nonlinear 
Markov evolution fit solving kinetic equation (I2.17p . Here we take B = 
C^iR'^) and M = P(R'^), so that B* = M^*3n(K) jg the set of Borel measures 
on R'^, so that M = sup^g-p(Rd) II/^IIb* = 1- 

Let y4[t,/i], t > 0, /i G P(R'^), be a family of operators in Coo(R'^) of the 
Levy-Khintchin type (11.120 . that is 

A[t, fj]f{z) = z, /i) V, V)f{z) + {bit, z, fx), Vf{z)) 

^ (3.1) 
(/(^ + y)- f{z) - iVfiz),y)lBMMt, z, fi, dy), 

such that each of them generates a Feller process with one and the same 
domain D such that ^^(R'^) CDC C^(R'^). 

Definition 3.1. A nonlinear Markov process on R"' specified by a family 
of generators {A[t,fi] : t > 0, fi E P(R'^)},0 < s < t is a family of pro- 
cesses {Xgt : fi G V(R'^)} defined on a certain filtered probability space 
(n, J-", {J-^}, 'P(R°')) that solve the nonlinear martingale problem, specified by 
the family {A[t, fi]}, that is for / G D, 

f{X^,) - f A[r, C{X^^;)]f{X^^^)dr, s < t (3.2) 

J s 
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is a martingale and C{X^^ = fi, where C{^) denotes the distribution of a 
random variable ^. 

Remark 3.1. If the operators A do not depend on t explicitly, i.e. A[t,fi\ = 
A[^], {Xg f} is a time-homogeneous nonlinear Markov process. If the opera- 
tors A do not depend on /i, i.e. A[t,fj] = A[t], {X^^} is a classical (linear) 
Markov process. If the operators A depend neither on t nor on fit, {Xg ^} is 
a time-homogeneous Markov process. 

Proposition 3.1. (i) For any nonlinear Markov process {X^^}, its marginal 
distributions fit solve fl2.17p . 

(a) Suppose the assumptions of Theorem \2.S\ are fulfilled with generators 
A[t,fi] of type lis. Then it is possible to construct a nonlinear Markov 
process {X^t}, with fit = C{Xgt) solving the Cauchy problem for equation 
fl2.17p with initial condition fi. 

Proof, (i) Since (13.21) is a martingale, 

Ef{X^,) = E/(X,^J + r A[t, C{Xi!^^)]Ef{X^^^)dr, s < t. 

J s 

Hence, for fit = C{X^t) we have 

(/, /^i) = (/, Ms) + / {A[t, fir]f, fir) dr. 

J s 

Consequently, we firstly conclude that fit depends continuously on t in D*, 
and secondly obtain (12.171) by differentiating this integral equation. 

(ii) By the assumptions of Theorem l2.2l a solution fit € 'P(R°') of equation 
(I2.17P with initial condition fig = fi specifies a propagator U^''^[fi], s < r <t, 
of linear transformations in B*, solving the Cauchy problems for equation 

^^{f,ut) = {A[t,fit]f,iyt). (3.3) 

In its turn, for any u G V(R'^), equation (13. 3p specifies marginal distributions 
of a usual (linear) Markov process {X^j(i/)} in R'^ with the initial measure 
u. Clearly, the process {X^^{fi)} is a solution to our martingale problem. □ 

Remark 3.2. The proof of statement (ii) suggests that one can look at a 
nonlinear Markov process as a family of solutions fit to a nonlinear kinetic 
equation, so that to each solution fit there is an attached usual (so-to-say. 
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tangent) Markov process, {Xg ^(u)}, specified by equation (\3.3\j . In Section\B 
we shall see how these tangent processes appear in the context of interacting 
particles as processes followed by tagged particles, see Remark \9.1\ 

Using martingales allows us to prove the following useful regularity prop- 
erty for the solution of kinetic equations. 

Proposition 3.2. Suppose the assumptions of Theorem \2.2\ are fulfilled for 
a (Markovian) equation of type fl2.17p with generators A[t,fi] of type ^2.17) . 
Let {X'^f} denote a nonlinear Markov process constructed from the family of 
generators by Proposition \3.1\ Assume, forp G (0,2] and P > 0, the 

following boundedness condition holds: 

sup max{|G(t,x,/i)|, |6(t,a;,/i)|, / mm{\y\^,\y\P)u{t,x, ^,dy)} < P, 

xeR'',t>0,fi€M J 

(3.4) 

and the initial measure fig = has a finite pth order moment, i.e. 

Then the distributions £(X^J = $*'*(/i), solving the Cauchy problem for 
equation (12.171) with initial condition /i^ have uniformly bounded pth mo- 
ments, i.e. 

J \x\P^'''{fi){dx)<c{T,P)[l + p^], (3.5) 
and are ^-Holder continuous with respect to t in the space {C npi^'^))* , i.e. 

_ $*2'^(^)||(^^^^(j,,)). < c(T,P)v/|ti-t2|, Vti,t2 > s > 0, (3.6) 
with a positive constant c. 

Proof. For a fixed trajectory {^t}t>o with initial value /i, one can consider 
{X^^} as a usual Markov process. Using the estimates for the moments of 
such processes from Section 5.5 (formula (5.61) of [54j (more precisely, its 
straightforward extension to time non- homogeneous case), one obtains from 
n that 

E [min (|X,^, - x\\ \X^, - x\^) \X^^, = x)] < e^(^'^)(*-) - 1. (3.7) 
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This implies ( I3.5p . Moreover, f l3.7p implies that 

E [\X^,t-m\x^,-,\<i\X^,s = x[< (e^(^'^)(*"^) - 1)1/2 < C(r,P)Vt^ 

(3.8) 

E [|X,^, - x|l|^M^_,|>i|X,^, = x\< (e^(^'^)(*-) - 1)Vp < C(T, P)(t - .)Vp, 

(3.9) 

and consequently 



E {\X^^, - x\ \X^^, = x)< C{T, P)Vt^, (3.10) 

where constants C(T, P) can have different values in various formulas above. 
Since $*''^(yu) is the distribution law of the process {X^ f-}, 

_ $^.*^(^)||(^^^^(j,,)^. = sup \E^PiX::,J - E^(X,^,J| 
< sup E|^(X,^,J-^(X,^,J| 
<E|X,^,^-X,^,J. 

(3.11) 

From (I3.10p . (13.110 and Markov property, we get (13. 6p as required. □ 

Remark 3.3. For the case of diffusions, Holder continuity ( 13. 6p was proved 
m [31]. 



Our main purpose for presenting Proposition 13.21 lies in the following 
corollary. 

Proposition 3.3. Under the assumptions of Theorem \2.1\ for generators 
{yU.}] of Levy-Khintchin type, but without locality condition (12. lip . sup- 
pose the boundedness condition (13.40 holds for some p G (0,2] and P > 0. 
Then the compactness condition from Theorem 2.J\ (stating that set (I2.20p is 
compact in V(R'^) ) holds for any initial measure fi with a finite moment of 
pth order. 

Proof. It follows from (13. 5 p and an observation that a set of probability laws 
on R'' with a bounded pth moment, p > 0, is tight and hence relatively 
compact. □ 
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4 Basic examples in and x {1, • • • , K} 



In this section, we present some basic examples of generators that fit to 
assumptions of the abstract theorems in Section [2j 

Notice that for the study of Markovian kinetic equations (without a cou- 
phng with control) the most nontrivial condition of Theorem 12.11 is (ii), as it 
concerns the difficult question from the theory of usual Markov process, on 
when a given pre-generator of Levy-Khintchine type does really generate a 
Markov process. More difficult is the situation with time-dependent genera- 
tors, as the standard semigroup methods (resolvents and Hille-Phillips-Iosida 
theorem) are not applicable. Further discussion of this topic and other ex- 
amples can be found in literature suggested in Section [Tnj 

Remark 4.1. For general anticipating equations coupled with control, as- 
sumption (i) of Theorem \2.1\ (yielding feedback regularity) becomes nontrivial 
as well. Verifiable conditions for this assumption to hold are given at the 
end of Section as the main consequence of the sensitivity theory for HJB 
equation developed there. 

Example 4.1. Nonlinear Levy processes are specified by a families of gen- 
erators of type ^3. 1\) such that all coefficients do not depend on z, i.e. 



The following statement is a consequence of Proposition 7.1 from [5^ 

(and can be easily obtained from the general theory of processes with inde- 
pendent increments as developed e.g. in l3U^ ). 

Proposition 4.1. Supposed that the coefficients G,h,v are continuous in t 
and Lipschitz continuous in fi in the norm of Banach space (C^(R'^))*, i.e. 



\\Git, fi) - Git, T]) \\ + \\b{t, /i) - b{t, r])\\+ / min(l, \y\^)Ht, /i, dy) - z/(t, r], dy)\ 



with a positive constant c, then condition (ii) of Theorem \2.1\ holds with 





<c||/^-^?||(c^(Rd))-, Vt>0, 



D = C^(R'^). 
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Example 4.2. McKean-Vlasov diffusion are specified by the following stochas- 
tic differential equation 



where drift coefficient b : x R*^ x V(R'^) — R*^, diffusion coefficient 
a : R"*" X R"' x P(R^) — )■ R^ and Wt is a standard Brownian motion. The 
corresponding generator is given by 



where G(t,x,^) = tr{a{t,x, iJ,)a^(t,x, ^)}. It is well known (and follows 
from Ito 's calculus ) that if the coefficients of a diffusion are Lipshitz contin- 
uous, the corresponding SDE is well posed, implying the following. 

Proposition 4.2. If G,b are continuous in t, Lipshitz continuous in x and 
Lipschitz continuous in /x in the topology of (C^(R°'))*, then the condition 
(a) of Theorem \2.1\ is satisfied. 

Example 4.3. Nonlinear stable-like processes (including tempered ones) are 
specified by the families 



Here s = y/\y\, K > 0, Ut are certain finite Borel measures on S'^ ^ and 
uifjX, fj,,dy) are finite measures, a, a are positive bounded functions with 



The following result is a corollary of (a straightforward time-nonhomogeneous 
extension of ) Proposition 4-6.2 of f^J. 

Proposition 4.3. If all coefficients are continuous int, a, a are -functions 
in x,s, b and v are Lipshitz continuous in x and fi (with fi taken in the 
topology (C^(R'^))*, then all conditions of Theorem \2.1\ are satisfied with 



dXt = bit, Xt, fit)dt + ait, Xt, fit)dWt 



A[t, fi]f{x) = [bit, X, f,), V/(x)) + ^{G{t, X, /x), V'fix)), f G CliR'), 




utids). 
(4.1) 



a e (0,2). 



D = C^(R'^). 
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Example 4.4. Processes of order at most one are specified by the families 



A[t,n]f{x) = {b{t,x,fi),Vf{x)) + / {f{x + y) - f{x))u{t,x,fi,dy), 

with the Levy measures v having finite first moment J |y|z/(t, x, /i, dy). The 
next result is established in Theorem 4-17 of [53]. 

Proposition 4.4. // b, v are continuous in t and Lipschitz continuous in fi, 
i.e. 



- 6(t,x,?7)||+ j \y\u{t,x,iJ,,dy) - iy{t,x,r],dy)\ 

<c\\^^-v\\{cu^'i)r, Vt>0,a;eR'^ 

and Lipshitz continuous in x, then condition (ii) of Theorem \2.1\ is satisfied 
with D = Ci,(R'^). 

The generators of order at most one describe a variety of well known 
models including spatially homogeneous and mollified Boltzmann equation 
and interacting a-stable laws with a <1. 

Example 4.5. Pure jump processes on a locally compact metric X are spec- 
ified by integral generators of the form 

A[t, fi]f{x) = [ {f{y) - /(x))z/(t, X, /i, dy) (4.2) 

jRd 

with bounded measures v. Assuming, unlike our basic assumption fll.141) . 
that we can control jumps of this process, i. e. measure v depends on control 
u, basic kinetic equation (11. lip takes the form 

^(/, ^^t'> = j j ^f^y) ~ fi^))^i'^^ ^' /^t' r(t, X, {/i>t}), dy)fit{dx), (4.3) 
or in the strong form 

—jitidx) = / z/(t, z, lit, r(t, z, {/U>t}), dx)iJit{dz) 

(4.4) 

- fit{dx) / u{t,x,iJ,t,T{t,x,{i2>t}),dz), 
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and the corresponding HJB equation becomes 



dV , , 
--—(if:, x)+max 

Ot u 



J{t,x, iJ,t,u) + / (y{y) -V{x))u{t,x,nt,u,dy) 



0. (4.5) 



In particular, if X is a finite set {1, ■ ■ ■ , K}, measures become K-dimensional 
vectors and kinetic equation (I4.4p rewrites as 

= ^{t,j, /ii, r(t, J, {/i>t}), X] ^> r(t, i, {/i>t}), j)- (4.6) 

/t is easy to see that, if measures v{t,x, ^f,u,dy) are uniformly bounded, the 
conditions of Theorem \2.1\ are satisfied with D = Coo(R'^)- For unbounded 
rates we refer to (and references therein) for a detailed discussion. 

Finally, we are interested in a multi agent setting of Subsection 11.31 

Theorem 4.1. Let A = {Ai, ■ ■ ■ , Ax) be a family of operators in Coo(R'^) 
of form (11.141) . with u = T(t, z, {fi>t}) , where Lj are given by one of the 
examples above, with all required continuity assumptions holding for each Ai . 
Moreover, let hi be a Lipshitz continuous function ofu and V depends Lipshitz 
continuously on all its arguments with measures considered in the topology of 
D*. 

(i) Then all conditions and hence the conclusions of Theorem \2.1\ are 
satisfied. 

(a) If boundedness conditions of Proposition \3.3\ hold for each Ai, i = 
1, ■ ■ ■ ,K, then the compactness requirement of Theorem 2.4 holds for any 
initial measure fi with a finite moment of pth order and hence the corre- 
sponding existence result holds. 

Proof. Looking at the proofs of all above statements, one sees that they 
extend straightforwardly to this multi-agent setting. □ 



5 Cauchy problem for HJB equations 

In this section, we are concerned with the general well-posedness of the 
Cauchy problem for the HJB equation (II. 9p . More precisely, we shall prove 
that there exists a unique mild solution V{t,x) to the Cauchy problem 

H,(x,VV) + L,V 




27 



with rather general Hamiltonian Ht, assuming that the propagator generated 
by Lf has certain smoothing property. Here we mostly follow Chapter 7 in 
|54j . preparing the setting for the sensitivity analysis of the next section. 

Remark 5.1. In the control theory , Hamiltonians appear in form fll.lSp . 
i.e. as 

Ht{x,p) = max{h(t,x,u)p + J(t,x,u)). (5.2) 

where u are controls, x,p G R'^ and t > 0. 

It is well known and easy to show (by Duhamel's principle) that if \^(t, x) 
is a classical (smooth) solution to fl5.ip . then it also satisfies the following 
mild form of (15. ip . 

V{t, x) = (f/*'^\/^(-))(x) + U^'^Hsi ■ , VV{s, ■)){x)ds. (5.3) 

This integral form is usually easier to analyze than the original equation (15.11) 
and is sufficient for applications to optimal control problems. Hence we shall 
deal here with the well-posedness of the mild equation. 

Theorem 5.1. Suppose 

(1) Ht{x,p) is continuous in t, Lipschitz continuous in p uniformly in x, 

i.e. 

\\Ht{x,p)~Ht{x,p')\\<Ci\\p-p'l p,p'gR^ VxGR^tG [0,T] (5.4) 
with a positive constant c\ and there exists a constant h > such that 

\\Ht{x,0)\\ <h VxgR"', tG[0,r]; (5.5) 

(2) the operators Lt generate a strongly continuous backward propagator 
11^''"^ in the Banach space Coo(R^) with a common invariant domain D C 
Cl^{'Rf'), and the subspace C^(R°') is also invariant, so that U^'^ are also 
strongly continuous in and 

W'^HCU^^) < C2, W'^^c^^^.) < C2 (5.6) 

for all t < s < T and a constant C2 > 0. 

(3) the operators f/*'"* map Coo(R'*) to C^(R'^) for t < s, and 

\\U'^'<P\\cu^<^)<wis-tm\c^^^.^, (5.7) 
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for t < s < T and for all G Coo(R'^), with an integrable positive function 
w on [0, T] . 

Then for any terminal data V^{-) G C^(R^), there exists a unique solu- 
tion V{t, x) to mild equation Ii5. 3\) . which is of class C^(R'^) for all t. 

Proof. We shall write sometimes shortly and Coo for C^(R'^) and Coo(R'^) 
respectively when no confusion may arise. Define an operator acting on 
C^^([0,T],C^(R'^)) by the formula 

M^]{x) = + ^ W'^H^i; V(Ps)ix)ds (5.8) 

We claim that ^ maps C^t([0, T], C^(R'^)) to itself. 
By the triangle inequality and (15. 4p . 

ll^*(^,V0,)llc^ < \mx,0)\\c^ + \\H,{x,V<l>t)-H,{x,0)\\c^ 
< h + ci\\V(j)t\\c^ 

From assumptions (2)- (3), 

ll^(0*)llc^ < ||t/*'^V^^(-)llc^ + ^^|t/''^i^*(-,V0.)||c^rf. 

<C2\\V''\\ci^ + j\{s-t)\\Hs{;VcPs)\\c^ds (5.9) 
<C2\\V^\\ci^ + {h + ci sup l|0s||c^) / wis~t)ds. 

t<s<T Jt 

Hence 

Furthermore, by equation (15. 4p and (15. 7p . 

ll**(0^)-^*(0')llc^<^^|f/*ii^.(-,V0i)-if,(-,V0^)]||c;i„ci. 

<^ cMs-t)\\V<Pl-V<Pl\\c^ds (5.10) 
< ci sup Ul - (plWci, / w{s-t)ds 

t<s<T Jt 
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for 0^,0^ G C^t([0, T], C^(R'^)). So the mapping is a contraction in 
C^t{[0, T], Cl^i'R'^)) for time tE[T-to,T] with to > small enough. Con- 
sequently \& has a unique fixed point for t G [T — to,^]- The well-posedness 
on the whole interval [0,T] is proved, as usual, by iterations. □ 

Remark 5.2. // each operator f/*'* has a kernel, e.g. it is given by 

U'''f{x) = J Git,s,x,y)fiy)dy (5.11) 

with a certain Green's function G, such that 

||V.G'(t,s,x,-)||Li(Rd) <«^(s-t) (5.12) 

then the smoothing condition ( [5. l\j holds. 

By the well posedness of equation (15.31) . its solutions define a propagator 
in C^(R'^). This implies, by the standard results (see e.g. [25); ^^lat these 
solutions are viscosity solutions to the original equation (15. ip . Moreover, for 
H of form (15.21) . these viscosity solutions solve the corresponding optimization 
problem. That is all we need for our purposes, allowing us to work further 
only with mild solutions constructed above. 

Remark 5.3. If we want a solution of mild equation / (5.3|) to solve the origi- 
nal HJB equation ( 15. ij) . one has to make some additional regularity assump- 
tions, see e.g. Theorem 7.8.2 of for detail. 

We would like to mention that Example 14.41 and Example 14.51 in Section 
m do not fit into our setting for the well-poseness of HJB equation (15. ip . 
since the smoothing property (15. 7p is hard to ensure for this examples. In 
the present paper we shall not develop an appropriate framework for HJB 
equation that would fit to these examples. 

Our main examples are summarized as follows. 

Proposition 5.1. (i) Under the assumptions of Propositions and \4 . 2\ 
suppose the matrix G is uniformly positive and is G^ as function x in the 
second case. Then the corresponding propagator (for any fixed {fJ't}ti^io,T]) has 
the smoothing property (15. 7p . (ii) Under the assumptions of Propositions^^ 
suppose the measure u has an everywhere positive density with respect to the 
Lebesgue measure on S'^~^. Then the corresponding propagator (for any fixed 
{fJ't}t<=[o,T]) has the smoothing property (15.71) . 

Proof. Statement (i) is well known in the theory of diffusion processes. State- 
ment (ii) is proved in [H], see also |5l]. □ 
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6 Sensitivity analysis for HJB equations 



For using nonlinear Markov processes as a modelling tool in concrete prob- 
lems (e.g. mean field games), it is important to be able to assess how sensitive 
the behaviour of the process is when the key parameters of the model change. 
Ideally one would like to have some kind of smooth dependence of evolution 
on these parameters. In this section, we shall discuss the sensitivity of solu- 
tions of HJB equations. 

Theorem 6.1. Suppose a continuous mapping {t, a) i-> L[t, a] from [0, T] xR 
to bounded linear operators L[t,a] : D Coo(R'^) is given, D C C^(R'^), 
such that 

(i) the linear operators L\t,a\ are differentiable with respect to a, i.e. 
dL/da[t,a] exist and represent a continuous (in t.a) family of uniformly 
bounded (by a constant ci > 0) operators D — )■ Coo(R''); 

(a) for any a G R, the operator curve L[t,a] : D ^ Coo(R^) generates a 
strongly continuous backward propagator of bounded linear operators U^f in 
the Banach space Coo(R'^) with the common invariant domain T>, such that, 
for any a, and t < s 

ma^m'/Wu, \\U'/\\c^, WU'/WcO < K (6.1) 

for some constant K > 0, and with their dual propagators t/*'* preserving the 
setV{R'^). 

(Hi) the operators U^f : Coo(R'') ^ C'^(R''), 0<t<s, and 

\\U'/<P\\cu^.)<w{s-tm\c^^n^), (6.2) 

for t < s < T and for all (j) e Coo(R'^); with an integrable positive function 
w on [0, T]. 

(iv) for any a G R, G D and dV^ /da exists in C^(R'^) and repre- 
sents a continuous mapping a >->■ C^(R'^). 
Then, for 

K.(^,-) = t/:'^e(-)eD, ViG[0,T] (6.3) 

the derivative dVa/doi{t, •) exists in the topology of the Banach space C^(R'^) 
and is uniformly bounded in C^(R'^). 

Proof. For notational simplicity, denote 

VL{t,x) = ^-^{t,x). 
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Differentiating both sides of the equation 



dt 

with respect to a yields 



(t, x) = —L\t,a]Va{t^x) 



d dL 
—V^{t, x) = -L[t, a]Vi{t, x) - —[t, a]\4(t, x) 



Then by the Duhamel principle, 

VLit.x) = Uf{y^)\x) + j^ U^\—[s,a]V^)Mds. (6.4) 

Since (V^)'(-) G Cj^(R°') and the family U^J^ is bounded as a family of 
mappings from C^(R'^) to Ci^(R°'), f/*'^K^(T, ■) belongs to C^(R°') and is 
uniformly bounded for bounded t, T. 

Also since for each pair (t, a) G [0, T] x R, the operator OA/ da[t, a] : D 
C^(R'^) and for < t < s < T, the operator U^^' : C^(R'^) ^ C^(R'^), we 
have 

Ul^\ — [s,a]V^){s,x)eCl{K'). 
From the assumptions (16. ip and (16.21) . the following inequality 

\\vL{t,-)\\ci^<\\u'f{v^y{-)\\ci^ 

"T or 

\U'f{g^[s,a]V^){s,-)\yjs 

^ 9L_._ , (6.5) 



r dL 

<K||V^||ci^ + nj{s-t)\\ — [s,a]V^{s,-)\\c^^^,)ds 

<(K + ci /" w{s-t)ds)\\V^\\j, 



gives us the uniform boundedness of the derivative V^(t, ■) in C^(R'^), show- 
ing thatK:(t,-)eC^(R'^). 

Finally, using smoothing property (16. 2p in conjunction with the argu- 
ments of Proposition 12.21 we can strengthen the statement of Proposition 12.21 
by obtaining continuous dependence of f/^'^ not only as operators in Coo(R''), 
but also as operators in C^(R'^). This implies the continuity of function 
(16.41) . as a mapping a C^(R'^). Together with our uniform bounds on 
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fl6.4p . this implies that this function does in fact represent the derivative of 
Va{t,x) with respect to a. □ 
Differentiating (16 .Sp with respect to a formally, we get 



K(t,.) = f^'^(vfy(.) + (^f^'^) 



(Vf)- (6.6) 



Our proof above and in particular formula (16 ■4p can be interpreted as saying 
that the derivative dU^^/da is well defined as a bounded operator D — >■ 
C^(R'=') such that 

(6.7) 

Theorem 6.2. Under the assumptions of Theorem \6 . 1[ suppose a family of 
Hamiltonian functions Ha^t{x,p) is given that satisfy, for each a G R, all 
the assumptions of Theorem \5.1{ with all estimates being uniform in a. As- 
sume moreover that the derivative dHa^t/da{x,p) exists and is a continuous 
function of all its variables. 

Denote by Vait,x) the mild solution of the Cauchy problem 

'^{t,x) = -L[t,a]Va{t,x) - Ha,t{x,VVa{t,x)) 



VJt=T = V. 



a 



constructed in Theorem \5.1\ Then the derivative dVa/da{t,x) exists and is 
uniformly bounded in the Banach space C^(R^) (as a function of x). 

Proof. For any a G R, by Theorem the unique solution V^(t,x) is the 
unique fixed point of the mapping ^a(0) defined by 

= U'fv;^ + U'fH^,s{., VU-))ds. (6.9) 

Differentiating with respect to a the fixed point equation = "^aiVa) and 
taking into account (16 .Tp we get the following equation 

K(t,.) = i^l + rL(K) (6.10) 

with 

U'f^{.,VV^{s,.))V^{s,.)ds, (6.11) 
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^i = ^ {u'fyl) + f (^^a^) HaA; vv;(., :))ds 

Hence formally, 

V^^ = (Jrf-r„)-^fi„, (6.12) 
where Id stands for the identity operator. By (16.110 . 



sup ||r.(<^)||ci, < / w{s-t) sup 

s£[t,T] Jt a,s,z,p 

< c{t) sup ll^.llc^ 

se[t,T] 



dp 



sup \\ips\\ci,ds 



with C7(t) — 7- as t — 7- T. Hence the inverse operator fl6.12p is well defined 
for t close enough to T yielding uniform boundedness of the l.h.s. of (I6.12p . 
Together with the continuity of in a, this allows us to conclude that the 
derivative V^{t, .) exists in for all t close to T. As usual, this statement 
extends to all t by iterations. □ 

Remark 6.1. Assuming some continuity int of the operators f/*'* : Coo(R'^) ^ 
Cl^(R'^), one could further claim that V^{t, .) is also continuous as mapping 
{a,t) H-> C^(R'^), but this does not seem to be crucial. 

We want now to apply these results to our basic mean field setting (in 
particular that of Subsection II. 3p . that is to HJB equation (I1.17p . (ll.lSp : 

dVUt x) 

—^ + Hl{x,VV\x),fit) + Ht,f^t]V'{t,x) = 0, (6.14) 

Hl{x,p,^t) := m.ax{hi{t,x, fit,u)p + Ji{t,x, fit,u)}. (6.15) 

This leads to the following crucial result. 

Theorem 6.3. Under the assumptions for W and Li from Theorem \6.2\ 
( recall that basic examples when smoothing property (16. 2p holds are given in 
Proposition \5. assume additionally 

(i) exists and is continuous and uniformly bounded; 

(a) exists in the topology o/D* = (C^(R'^))* and represents a con- 
tinous family of uniformly bounded operators D — )■ Coo(R'^); 
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(in) the unique point of maximum in fl6.15p is continuous in t and Lip- 
schitz continuous in {x,p,fi), uniformly with respect to t, x and hounded p 
and fi (again /x in the topology ofD*). 

Then, given a trajectory {fi.} e C^{[0,T], MiB*)) , M = V{R'^), and a 
final payoff , the feedback control u = T(t,x, {yU>t}) defined via equations 
fl6.14p and (16. 151) . is Lipschitz continuous in {fi.} uniformly, i.e. for any 
{^.}eC,i[0,T],Mm) 

\\T{t,x,{r]>t}~T{t,x,{^>t}\\<h sup ||r/.-es||D-, Vt G [0, T], x G R'^ 

se[t,T] 

with some constant ki > 0. 

Proof. Theorem 16.21 shows that the depends Lipshitz continuously on 
{/i.}. Taking into account other assumptions we conclude that the unique 
point of maximum in the expression 

max{hi(t, X, /it, u)W\t, x, {fi>t}) + Mt, x, fit, u)} 

has required properties. □ 

Remark 6.2. Lipschitz continuity of the resulting control T with respect to 
the parameter {/i.} was called the feedback regularity property in the pioneer- 
ing paper [37^ , where it was taken for granted as an additional assumption 
without verification. Our result above justifies this assumption of j!T7] by 
proving it in our much more general setting. 

Theorem 16.31 gives verifiable conditions for the assumption (i) of Theorem 
l2.1l to hold, which (together with compactness criterion from Proposition 13.31 
feeding in Theorem 12. 4 p allows us to establish local well-posedness and global 
existence of solutions to fll.lip in our basic setting of agents with a spatial and 
discrete component. This completes tasks Tl), T2) from the introduction. 
From now on we turn to the iV-particle approximations to the coupled mean- 
field evolution given by fll.lip . For notational simplicity, the arguments and 
results in the following sections will be presented for one class agents, i.e. 
K = 1. The extension to arbitrary K is straightforward. 
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7 Smooth dependence of nonlinear processes 
on initial data 



This section makes preparation for the analysis of the convergence of A'^ 
particle approximation in Section El 

Our aim is to prove smoothness of the solutions fit to kinetic equation 
f l2.17p with respect to initial data /io- In fact we need the existence and 
regularity of the first and second order variational derivative of fit with resect 
to fiQ. As a consequence, we are going to prove that the propagator of linear 
transformations on functionals of measures of the form 

($^'*(F))(/i,):=F(/i,(/i,)), 0<s<t (7.1) 

(where fitifJ's) denotes the solution of the kinetic equation f l2.17p with the 
initial condition ftg at time s) preserves the set of twice differentiable func- 
tionals. This fact eventually would allow us to compare the propagators of 
A^-particle processes with propagator (17. ip of the limiting deterministic pro- 
cess via Proposition 12.11 with D being the space of functionals with sufficiently 
regular second order variational derivative. 

We shall follow the line of arguments from [52] • However, essential simpli- 
fication and, at the same time, improvements will be achieved, in particular, 
by extending the results to time dependent family of generators and, most 
importantly, by weakening the requirements on the regularity of the coef- 
ficients of Levy-Khintchine type operators (13. ip with respect to position z, 
which is of principle importance for applications to MFG. 

Let us start with an abstract kinetic equation (I2.17P in a dual Banach 
space B* satisfying the assumptions of Theorem 12.21 Given a family of initial 
data /Xq depending on a real parameter a, we are interested in the derivative 

er = |^, w>o (7.2) 

of the corresponding solutions /x" with respect to this parameter. 

Differentiating (I2.17P (at least formally for the moment) with respect to 
a yields the equation 

|(/,er) = (A[t,/.^]/,c) + {v^^A[t,f,'^]f,f,t) (7.3) 

with the initial condition 

(7.4) 
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where 

V^A[t,f^]= lira -{A[t,fi + s^]-A[t,fi]), G B* (7.5) 

s— >-0+ S 

denotes the Gateaux derivatives of A[)f:,/i], where it is assumed that the def- 
inition of ^[t, /x] can be extended to a neighborhood of Ai in D*. 

The crucial point for what follows is the observation that one cannot 
expect equation (17. 3p to be well-posed in B* making usual results from the 
theory of vector-valued ODE unapplicable. Thus we are aiming at solving 
it in D*, so that the derivatives with respect to initial data are going to 
live in a different space than the solution itself. Moreover, even for solving 
equation (17. Sp in D* (as is done in [51], [57] for time-homogeneous family 
requires strong smoothness assumptions on the family 74[t,/i], which 
are not appropriate for analyzing MFG. Hence our plan is to solve (17.31) in 
a mild form, and then show that the solution does represent the derivative 

(ED- 



Assumption 7.1. The Gateaux derivative ( [y.5| ) exists in the norm-topology 
D* and is regular enough so that, given fi G B*, the bilinear form (V^A[t, fj]f, ii) 
on the arguments C, G D* and / G D, from (17. 3p can be written in the form 

iV^A[t,fi]f,fi) = i<l)tW,0 (7.6) 

with a bounded linear operator (ptlfA : D — t- D. 

We shall see later that this assumption is satisfied for the case where 
A[t, /i] is of Levy-Khintchine form (13.11) with coefficients depending smoothly 
on n (but, what is important, not necessarily so on positions z). 

Thus weak forward equation (17.31) rewrites as 

^(/,er) = (A[t,/i,"]/,er) + {Mf^?]f,m, (7.7) 

which is dual to the backward equation in D of the form 

|/r = -A[t,/i^]/,"-0,[/i^]/f. (7.8) 

Let us stress again that the operator (t>t[fi] acts from D — D (and is not 
defined on B). 

As in Section [5l assuming the family y4[t,/i"] generates a backward prop- 
agator f/*''', t < r, in B with the common invariant domain D, with the dual 
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forward propagator U^*, we can write the mild forms of equations f l7.8p with 
a terminal condition and fl7.7p with an initial condition respectively as 

ft = u'/f:^+ u'/<j>s[^^':]f:ds, t<r, (7.9) 

C = f>«'*er + f K'^MC ds, t < r, (7.10) 



and 



or in the weak form 

= (f/*'7,C) + l\Uf^':P'/-f,C) ds, feu. (7.11) 

where 4>l[fj] is the dual operator of (pt[^j]- As we mentioned, we are going to 
solve these integral equations not insisting on the fact that their solutions 
are related in any way to the original equation 07.71) . 

It is clear (and comes from the standard arguments of perturbation the- 
ory) that equations (17. 9p and (17.100 have unique solutions F^^ fr ^^cl F^^^t, 
where the propagators F^^ and its dual FJ^'* are given by the convergent (in 
D and D* respectively) perturbation series 

oo „ 

rf = U'/ + Y, / f^'^^0s.f/:^'^^---0s.f/r''^rf3i---ci3^, (7.12) 

m=l •'t<si< - <Sm<r 



oo „ 

= ill:' + / iJ:;'-4>sJ:—' ■ ■ ■ KK'^' ds,--- dsr. 

m=l J t<si<---<Sm.<r 

(7.13) 

where (ps = 4>s[f^"]- 

Theorem 7.1. Under the assumptions of Theorem \2.2^ let 

^° " da 

be defined in D* for a certain family of initial data /Iq ( with the derivative 
existing in the norm topology ofT)*). Moreover, let A[t,fi] depends on jj, 
smoothly in the sense that representation (17.61) holds with a continuous in 
t family of hounded linear operator (ptilA '■ -D D. Then fif is Lipshitz 
continuous in a in the topology D*, the r.h.s. of (17. 2p exists for almost 
all a (the derivative taken in the norm-topology ofT)*) and equals almost 
everywhere to the unique solution = C,t{^o,f^o) equation ( ]7.10p given 
by (EH- 
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Proof. The main idea is to approximate by bounded operators, use 

the standard sensitivity theory for vector valued ODE and then obtain the 
required result by passing to the limit. 

To carry our this program, let us choose a family of bounded operators 
A[t,fi]{n) : B — )■ B and a family of bounded operators (/)t[/i](n) : B — > B, 
n = 1, 2, that satisfy all the same conditions as A[t, /i] and and such 

that 

\\{A[t,fi]{n)-A[t,fi])g\\B^O and \\{<j)t[f^]in) - <f)t[fi])g\\T, ^ 

for all (7 G D. We shall use the same notations for corresponding solutions 
adding dependence on n for all objects constructed from A[t,^]{n). 

As such approximations for /i], one can use either standard losida 
approximation (which is convenient in abstract setting) or, in case of the 
generators of Feller processes, the generators of approximating pure-jump 
Markov processes. 

By the standard theory of ODE in Banach spaces (see e.g. [67J or Ap- 
pendices D,F in [S2]), the equation for /if (n) and C,t{n) are both well posed 
in the strong sense in both B* and D*, and C,"{n) represent the derivatives 
of /i"(n) in both B* and D*. Consequently 

/if (n) - /if"(n) = [ ef (n) dfi Wa, ao e R (7.14) 

holds as an equation in D* (and in B* whenever G B*). 

By the perturbation theory applied to nonlinear kinetic equations (see 
[52] . Section 2.1 or ^7\), fit{n) converge to /if in the norm-topology of D*. 
Moreover, from the perturbation series representation (17.131) . we deduce the 
convergence of C.tin) to in the norm-topology of D*. Hence, we can pass 
to the limit n — )■ cxd in equation f l7.14p in the norm topology of D* yielding 
the equation 

(7.15) 

This implies the statement of the Theorem. □ 

Remark 7.1. To establish continuity of ^'^ in a, one seems to require much 
stronger regularity assumptions (see Remark \7.2\ for these assumptions). And 
without this continuity one can get the derivative of the r.h.s. of fl7.15p 
only almost everywhere. Fortunately, for variational derivatives that we are 
aiming at, we are able to circumvent this difficulty. 
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Let us turn to the Levy-Khintchine type generators 



A% fi]f{z) = -{Git, z, //) V, V)/(^) + {b{t, z, fi), Vf{z)) 
+ f ifiz + y)- fiz) - iVfiz),y)lB, iy)Ht, z, dy), 



(7.16) 



in B = Coo(R'^) such that each of them generates a Feller process with one 
and the same invariant core D = C^(R'^). In order to do differentiation of 
the operators with respect to fi, the operators yu] should be well 

defined on a neighbourhood of P(R°'). So we introduce the set 



M 



{Mx :^ XViR'^y, Xi < X < X2} 



with < Ai < 1 < A2. 

We also need the following notations. Let C^^^(R"') denote the subspace 
of functions F{x) from C^(R'') such that the elements of 2nd derivative 
d'^F/dx^ are defined and belong to C^(R'') as functions of the second vari- 
able, equipped with the norm 



IcSr'CR'^) ll-^llcg.CR'i) + sup 



dxidxjdykdyi 



Let C*oo (-^(Ai.Ag)) denote the subspace of functionals F{jj,) from C(A^(Ai,a2)) 
such that the 1st variational derivative SF/SiJ,{x) is defined and belongs to 
C^(R'^) as a function of x uniformly in fi. This subspace becomes a Banach 
space when equipped with the norm 



l^llc^'(>I(Al,A2)) 



sup 

IImII<i 



6F 



Similarly we define the Banach space C'^^(A4(Ai,A2)) of functionals with the 
norm 

S^F 



l-^llf-^.a^A/f ^ '■— sup 



and C'to^^^(A^(Ai,A2)) of functionals with the norm 



sup 



S^F 
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It turns out (see Theorem 17. 41 below) that the space C'^^^^(A^(Ai,a2)) (and not 
a more natural space C^^(A^(Ai A2))) forms an invariant core for propagator 

The Gateaux derivatives f l7.5p are often specified via variational deriva- 
tives 

^-^^ = Vs^A[t,f,] = hm -{A[t,fi + s5.]-A[t,f^]). (7.17) 

By formal differentiation, as above, we expect the variational derivatives 

^t(x, /io) = r X = lim -(/it(/io + s5^) - fitifio)), (7.18) 
dfio{x) s->o+ s 

if they exist, to satisfy the mild version of equation 
dt J^d V J^d Sfit[v) ) 

(7.19) 

where 

^/(.) = \ { '-£^v, v) /(.) + ( vm) 

+ / (/(.+!/) - f{z) - (v/(.),y)iB,(i/)) ^^^';y:^^^ 

When there is no ambiguity, we sometimes write ^t(x) = C,t{x,fio). Equation 
f l7.19p naturally rewrites in form (17. 6p with 

(MMiv) = I ^-^^f{z)^,{dz). (7.21) 

Theorem 7.2. Let B = Coo(R'^), D = C^(R'^), M = Mx for each 
A e (Ai,A2), ^[t,/^] be of the Levy-Khintchine form (17. 16p . Suppose 
(i) for any {//.} G ^^([0, T], MxiY)*)) , the operator curve A[t, {ji}] : D h-). B 
generate a strongly continuous backward propagator of bounded linear oper- 
ators U^''^[{^Y\ m B, < t < s, on the common invariant domain D, such 
that 

||?7*'^[{/i.}]||D^D < ci and ||f/*'^[{/i.}]||B^B < C2, t< s, (7.22) 
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for some positive constants Ci,C2, and with their dual propagators 
preserving the set Aix; 
(a) the coefficients 

G{t, z, ■), b{t, z, ■), iy{t, z, ■) G C'J{M^x,M)) (7-23) 

uniformly in t, z as functions of fi, with uniformly bounded derivatives. 



Then the variational derivatives ^t{x) of the solutions fit = f^tifJ'o) with 
respect to fiQ are well defined as elements o/D* and are twice continuously dif- 
ferentiable in x (again in the topology ofT)*) with uniformly bounded deriva- 
tives. Moreover, they represent unique solutions of the mild forms of equation 
f l7.19p with the initial conditions C,o{x) = 6^. 

Proof. By the arguments given before Theorem 17.11 the solutions ^t(x, /xq) 
to the mild form of equation fl7.19p are uniquely defined in D*. Moreover, 
as ^j(x) solve linear equations, their derivatives with respect to initial data 
C,o{x){= 6x) solve the same linear equations. Therefore, since 

^eiCU^'W and ^ G D* = (C^(R'^)r, 
the derivatives 



dx dx"^ 

are well defined in D* and solve the same equation as ^t{x,fio) do. 

It remains to establish that ^t(x) do represent variational derivatives. To 
this end, let us follow the same argument as in Theorem 17.11 Approximating 
A[t, fi] by bounded operators A[t, /i](n) and passing to the limit in the integral 
equation defining variational derivatives 



F{Y + 4) - F{Y) = [ 

Jo 



' 6F{Y + s6,) , 
- as 



6Y{x) 

(see e.g. (F.l) in [52]) leads to the equation 

Ait(/^o + Sec) - IJ'tilJ'o) = / ^t{x, Ho + s6^) ds. 

Jo 

Unlike equation (I7.15P above, the functions on the r.h.s. are continuous, 
which allows us to deduce the required property of ^t(x). □ 
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Remark 7.2. ^45 C,t{x) play a key role in our exposition, some further de- 
tails seem to be in order. Firstly, if the propagator generated by A[t,fi] has 
smoothing property (15.71) and moreover, A[t,n] depends on fi only via the 
drift coefficients (that is Li from (I1.14p does not depend on fi, as is the case 
in the pioneering papers fSl]/ and l62]), then the propagator resolving equa- 
tion (17.91) ( or more precisely, the mild form of the equation dual to (17.191) ) 
acts by bounded operators m B = Coo(R''). Hence in this case the variational 
derivatives ^f(x) belong to the same space B* = M''*^'^(R'^) as the evolution fit 
itself making all continuity arguments much more transparent. Moreover, in 
the most of examples one can even show that the measures ^t{x) have density 
with respect to Lebesgue measure, i.e. they are represented by functions from 
L^(R'^). This situation was described in detail in /37]/ . Secondly, only as- 
suming that the propagator generated by A[t,fi] has smoothing property (15.71) 
but also extended to smooth functions, i.e. it also satisfies 

\\U''^ncu^.)<w{s-tm\cun^), (7.24) 

then, as one easily sees, the propagator resolving the mild form of the equation 
dual to (I7.19P acts by bounded operators in C^(R''). Hence C,t{x) belong to 
the space (C^(R'^))* and continuity in this space of the integrand on r.h.s. 
of (I7.15P is easily established. 

Let us analyze similarly the second variational derivative 



r]t{x,y) =r]t{x,y,fio) 



5fio{x)5fio{y) 



(7.25) 



Differentiating (again formally in the first place) equation (I7.19p . yields the 
equation 

±(f,rft{x,y))= [ (A[t,fx]f{v)+ [ ^-^M.f(^,)^^(^dz))rftix,y,dv) 

+ / (^/w + ^/w 

+ / '^.'^.^^'^j f{z)fit{dz))^t{x,dv)^t{y,du) 

(7.26) 
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with ?7o(x, y) = 0. Following the similar argument to the first order derivative, 
we shall look at the solution to f l7.26p only in the mild version: 



7]t{x, y) = f/*'%o(x, y)+ f f/*'^(0:[/i.]r/.(x, y) + ^(x, y)) ds, (7.27) 







where 4>*[fj] is the dual operator of 0s [/i] given in (17.2 ip and Gg G D* given 
by 



J^2d \ dHs{u) dfis{v) 



+ [ giz)fisidz))^six,dv)^siy,du). 

(7.28) 

For equation (I7.27P to make sense, we need only to make sure that the 
last term in ([72HD is well defined for e D* = (^^(R'^))*. This leads to 
the following extension of Theorem 17.21 to the second variational derivative. 



Theorem 7.3. Under the assumptions of Theorem \7.2\ suppose additional 
regularity on the coefficients, namely 

Git, z, ■), &(t, z, ■), u{t, z, ■) G C7^'"'(>1(A„A,)) (7.29) 

uniformly in t, x as functions of fi, with uniformly bounded derivatives. Then 
the second variational derivatives rit{x,y) of the solutions fit = yUt(/io) ''^ith 
respect to fiQ are well defined as elements of D* and are twice continuously 
differentiahle as functions of x and y (again in the topology of Y)* ) with 
uniformly hounded derivatives. Moreover, rit{x,y) represent unique solutions 
of the mild forms of equation (I7.26P with rio{x, y) = 0. 

Proof. It is almost straightforward extension of Theorem 17. 2[ However, dif- 
ferentiating equation fl7.26p with respect to x and y, one has to take into 
account that dependence on x, y is not only via initial conditions, but also 
via the r.h.s. containing ^t{x,dv)C,t{y,du). This product form allows to dif- 
ferentiate twice in x and additionally twice in y, as need to get a function of 
class C'J-'iM^,,,x,))- □ 

Remark 7.3. // G, b, v depend on fi as smooth functions of some multiple 
integrals (which is usually the case in application) of the form 

gixi,--- , Xn)lJ^{dxi) ■ ■ ■ n{dxn) 
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with a smooth function g, then all variational derivatives are defined and are 
smooth. Hence smoothness of functions fl7.23p and fl7.29p are very natural 
and not much restrictive assumptions. 

Remark 7.4. In case A[t,^] = A[t] not depending on ^, i.e. in case of a 
usual Markov process, ^((x, /iq) does not depend on fiQ and coincides with the 
transition probability P{t,x,dy) of this process. Moreover, T]t{x,y) vanishes. 

Finally we are able to analyze the transformations generated by the flow 
fiQ = fi fit on functionals, that is the transformations = 
F{fit{fi)), or the corresponding backward propagators (17.11) . 

These transformations are well defined as linear contractions on the space 
C(A^(Ai,A2)) of continuous in D* functions on A^(Ai,A2)5 since 

||$^'*(i^)||c(A^(,,,„)= sup |<f^'*(F)(/x)| 

= sup \F{fit{fis))\ < sup (7.30) 

MG>!(Ai,A2) Me>!(Ai,A2) 

= ll^ll'^(^(Ai,A2))- 

Theorem 7.4. (i) Under the assumptions of Theorem 7.2, the transfor- 
mations (17. ip form a backward propagator of uniformly bounded (for s,t 
from any compact interval) linear operators in C^^(A^(Ai,A2)) ■ ('^^j Under 
the assumptions of Theorem \7.3\ the transformations (17. ip form a backward 
propagator of uniformly bounded (for s, t from any compact interval) linear 

^(Ai,A2))- 



operators in C^^^^(A^( 



Proof. It follows from Theorems 17.21 and 17.31 and the formulas 
(5$0'*(F))(/x) _ 5F(/i,) _ f 5F{fit) 



(^^$°'*(F))(/i) _ 6'F{fit) 
6fio{x)5fio{y) Sfio{x)5fio{y) 
6F{fit) 



r , ^ Vtix,y,fio,dz) (7.31) 
+ / ^f'^S^^] M {x,fio,dz)Ct{y,f^o,dw). 

□ 
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8 Convergence of A^-particle approximations 
(dynamic law of large numbers) 

Here we start with the refining of some results on the weak LLN hmit for 
interacting particles from Chapter 9 of |52j based on the improvements of the 
regularity results achieved in the previous section. Then we extend them in 
two directions: we prove that all the rates of convergence remain valid (i) if 
one of the agent would be allowed to deviate from the common behavior and 
(ii) if one uses integral functionals on measure paths (interpreted as integral 
payoffs in the game setting), rather than just on marginal measures. 

To this end, let us return to operator Af^ of form f ll.3l) from the intro- 
duction (where we omit, for the moment, the dependence on u and 7): 



N 



A^F{6^/N) = Y,A%f^]f{xi, 

1=1 

where /(x) = F(5x/^), x = (xi, ■ ■ ■ ,xn) and 



1 ^ 1 

i=l 



Assume now that v4[t,yu] is of the Levy-Khintchine form (17.161) in B = 
Coo(R'^) and such that each generates a Feller process with invariant 
core D = C^(R'^^). The following calculations are performed in Chapter 9 
of 1521. 



Proposition 8.1. If F is smooth enough, which means that all the deriva- 
tives in the formulas below are well defined, then 



AfF(/i) 



A[t,fx] 



+ 



N 



[l—s) ds 



6F 
S^F 



d d 

W + ^ ( G(t,x,/x)— , — 



\ 6^F 




1 6fi{x)6fi{y) 


y=x_ 



ji{dx) 



where ^ = dy^/N . 
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This formula yields a precise estimate for the deviation of operator flS.ip 
from the limiting operator 

6F- X (8-3) 



Remark 8.1. If the junctionals F are linear, i.e. of the form (11.41) .• Ft{^) = 
{f,IJ,) for a f e C(R'^), then the r.h.s of ([Hj]) is the one of (ifTTD (with 
fit = /ij. 



According to Theorem \7.A\ this family of operators generates propagator 
(17. ip on the common invariant domains C'^^(A^(Ai,a2)) ot C'^^^^(A^(Ai,A2))- 
Thus we need F G C^^(A^(Ai A2)) to have AfF to be well defined and F e 
(Ai.Aa)) to have an estimate of order 1/N for the deviation of AjF 
from A^F. 

Let f/^*, s < t, denote the backward propagator on C {V^ (R'^)) generated 
by (18. ip and is defined in (17. ip . From Proposition 12. H we have: 

[{U'/ - «1>^'*)F](/.) = r[f/r(l^ - A,)$^'*F](^) d.s, (8.4) 

for F e C^^^'^iM^x^M)) and fi G Pi^(R'^). By Theorem O and Proposition 
18. H equation (18. 4 p implies 

sup lim' - <^''')Fm < ^{t - .)l|Fl|^..x.(^ (8.5) 

for < s < t < T, with some constant C (T) arising from the bounds on the 
propagators on C'^^^^(A^(Ai,a2)) from Theorem 17. 4[ 
This leads to the following result. 



Theorem 8.1. Under the assumptions of Theorem \7.3[ suppose the initial 
conditions ^ 

converge in D* = (C^(R'^))*, as N 00, to a probability law fio G V(R'^) 
so that 

||/io^-/io||D. (8.6) 
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with a constant Ki > 0. Then, for t G [0, T] with any T > 0, 

\[U'/F]{f^^) - [$°'*(F)](/.o)| < + (8-7) 

with a constant C(T). 
Proof. For F G C'^^''^(A^(Ai,A2)), we have 

Estimating the first term by (18. 5p and the second one by the Lipshitz con- 
tinuity of the solutions to kinetic equation obtained in Theorem 12.21 and 
assumption fl8.6p . yields f l8.7p . □ 

Let us turn now to full operator (11. 3p with u and 7 included. Assume 
that all agents follow some fixed strategy 7(i, x), but for one player, say 
the 1st one, who applies a different control Ui^t = 7(^52;). Then, instead 
of the operator (18. ip . the controlled process of interacting agents will be 
generated by the operator 

=A^[^]Fi5jN) + [A\t,^^,^it,.))-A\t,^^,^it,.))]Fi5JN), 
where /z = 6JN, F G C^'"'(A^(Ai,a.)) and 

N 

Af[^]FiSjN) = ^ A% /i, 7]/(xi, • • ■ , x^). 

i=l 

Remark 8.2. // the agents only control their drifts, as we usually assume 
here, i.e. generators A^[t,fi,u] has the form of (I1.16p . then 

[A\t, 7(t, .)) - A\t, /i, 7(t, miSjN) 
= {h{t,fi,^{t,.))-h{t,fi,^{t,.)),VF{6jN)). ^ ■ ' 

It is clear that the result of the application of the last term of (18. 8p yields 
an expression of order 1/A^, allowing us to conclude that 

Ani,l]F{5jN) = If [7]F(5JiV) + 0(1/Ar), 
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where 0{1/N) is uniform for F from bounded subsets C'^^^^(A^(Ai,a2))- Hence 
[7,7]F and [7]^ have the same hmit At[7] defined in (18.31) . and the 
same rate of convergence of order holds for F from bounded subsets 
(Ai,A2))- However, instead of just comparing the marginal distri- 
butions of Markov processes /if^[7,7] specified by the propagators U^^[y,^] 
(and generated by the A^[^,^]), as in Theorem 18. ![ we also need to be able 
to compare their integral characteristics (interpreted as payoffs in a game 
theoretic setting). 

The following result is a simplified preliminary version of Theorem 19.11 
below. 



Theorem 8.2. Suppose the assumptions of Theorem \7.3\ hold for the family 
of operators A[t, /i, •yit, .)] with 

A[t, fi, u]f{z) = hcit, z, fi, u)V, V)f{z) + {b{t, z, fi, u), Vf{z)) 

. (8.10) 
+ J {f{z + y)- f{z) - {yf{z),y)lB, {y)Mt, z, /i, u, dy), 

and a class of functions 7 : x R'^ — )■ U, that are continuous in the first 
variable and Lipschitz continuous in the second one. 
Suppose the initial conditions 

Un = —(SxN + ■ ■ ■ + SxN ) 

converge in D* = (C^(R'^))*, as N ^ 00, to a probability law /io G VilV^) 
so that 

ho -f^oWo' < 

with a constant ni > 0. 

Let f/^*[7,7] denote the propagator generated by Af^[7,7] and ^^'^[y] the 
propagator generated by At [7] . 

(i) Then, fort G [0,T] with any T > 0, 

C{T)f,^, \ (8-11) 

with a constant C{T) not dependent on 7. 
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(zz) Moreover, let J{t,ij,) e C{[0,TIC^^''\Mx,m))- Then 



E 



< 



C{T) 



J{s,fis[l]) ds 



(8.12) 



(T 



J\\c{[0,T],cl^^^\Mx^,x^)) + '^1 



where /if' [7, 7] is the Markov process specified by the propagators ?7^*[7,7] 
and ^it[l\ is the solution to kinetic equation (11.71) with the initial /io- 

Proof, (i) This follows from (18. 8p by the same argument as in Theorem 18.11 
(ii) Let us represent the integral J(s, /i^[7, 7]) ds as the limit of Rieman- 
nian sums. Then (18.121) is obtained by applying (18. lip term-by-term and 
passing to the limit. □ 



9 Mean field limit as an e-equilibrium 

In this section, we complete our program by performing task T4), that is by 
showing that a solution to equation (11. lip provides a perfect e- equilibrium for 
the corresponding approximating game of players, with e of order 1/N. 
For simplicity, we consider one class of agents. Extensions to multi-class 
agents, using Theorem 16.31 is straightforward. 

Let us recall, that for e > 0, a strategy profile F in a game of N players 
with payoffs V^(F), i = 1, ■ ■ ■ , A^, is a e-equilibrium (or e-Nash equilibrium) 
if, for each player i and an eligible strategy Ui, 

v,{r)>v,{r.„u,)-€, 

where (F_j, Ui) denotes the profile obtained from F by substituting the strat- 
egy of player i with Uj. A profile of dynamic strategies in a dynamic game on 
a time interval [0, T] is called a perfect e- equilibrium, if it is an e-equilibrium 
for any subgame started any time t G [0, T] (see a systematics presentation 
of this notion in [58j or any textbook on game theory, original papers on 
perfect e-equilibria being and [2B])- 

To complete our program, we need to extend the estimate (18.120 to the 
case of J depending on the position of a tagged agent and its own strategy. 
To this end, let us look at the process of pairs {Xf^, /if^), that is on a chosen 
tagged agent and the overall mass. 



50 



Let C^^'^^^(R'^ X A^(Ai.A2)) denote the subspace of functionals F{x,^) 
from Coo(R'^ X M (Ai,A2))) the space of continuous bounded functionals on 
R'^ X A^(Ai,A2)) such that for each /i G A^(Ai,A2)) the functional F{.,fi) G 
CliR'^) and for each x G R^ F{x, .) G C^'^'('-M(Ai,A2))- 

Let C([0,T] X W,C^2,2x2^p^d ^ M(^XuX2))) denote the subspace of func- 
tionals F(t,x, fi,u) from Coo([0,T] x R"' x A^(Ai,A2) ^ the space of con- 
tinuous bounded functionals on [0,T] x R'^ x A^(Ai,A2) ^ such that for 
each (t,fj,,u), the functional F(t,.,fi,u) G C^(R°') and for each {t,x,.,u), 
Fit,x,.,u)EC'J>^'{Mix,,x,)). 

The generators of the process of pairs {Xf^^, nf) are defined on the space 
^2;2,2x2(j^d X Mi^x^M)) ^nd take the form 

Ai4l,l]F{x„5jN) := {A%5JN,^]+Af[^,^]) F{x,,6jN), (9.1) 

where the 1st (resp. 2nd) operator in the sum acts on the 1st (resp. 2nd) 
variable of F [A^ just means the operator A acting on the variable Xi). 
Consequently, by equation (18.81) and Proposition 18.11 

A^Msb, l]Fix^, 6JN) = {A% 7] + A47]) Fix,, 6JN) + 0{l/N). 

(9.2) 

Theorem 9.1. Under the assumption of Theorem \8.2\ let U^^.^^glj , j] denote 

the propagator generated by A^^^g[y,'y] and ^tagil] propagator generated 
by the family 

A%fi,^]+At[^] 

where ji = S^/N. Suppose additionally X^q G R'* converge, as N 00, to a 
point Xi Q G R'^ such that 

ll^fo-^i,o||<| (9.3) 

with a constant K2 > 0. 

(i) Then, for t G [0, T] with any T > 0, 



C(T) / 

with a constant C{T) not dependent on 7. 



(9.4) 
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E 



< 



(ii) Moreover, ifj{t,x,n,u) G C{[0,T]xU,C^^''^'''^{R'^ x M(^x,,x,))) , then 
^ J(s,Xi^„/if[7,7],7(s,Xi^J)ds-E^ J{s,X,,s,Mlis,X,,,))ds 



c 

N 



(9.5) 

where the pair (Xf„/if [7,7]) is the Markov process specified by the propa- 
gators ^Ar*ta3[7;7] A^sW is the solution to kinetic equation ( \1.7\\ with the 
initial fi^ and the Markov process Xi <, is generated by A^[t, fit['y],'y]. 

Proof, (i) For F G C^^''^'''^{R'^, M 



(Ai.Aa);, 
N ,.N 



we have 



< 



0,t 



+ 



0,t 
tag I 



.iV^ 



(9.6) 



[<|.^;[7]F](Xi,o,/i;r) - K,[7]F](Xi,o,/xo 



Estimating the first term by (18.51) . the second one by assumption (19. 3p and 
the third one by the assumption (18. 6p and the Lipschitz continuity of the 
solutions to kinetic equation obtained in Theorem 12.21 yields (19. 4p . 

(ii) Let us represent both integrals J(s, Xf^, /if^ [7, 7] , 7(5, X^^)) ds and 

/^"^ J(s, Xi^s, /is[7], 7(5, Xi^s)) (is as the limits of Riemannian sums. Then 
(19. 5p is obtained by applying (19. 4p term-by-term and passing to the limit. □ 

Remark 9.1. Theorem \9. 1\ remains valid if the initial points Xf^Q are random 
and their distributions converge to a certain law v. If = 7, the limiting 
process Xi ^ is the tangent Markov process to the nonlinear process fit, see 
Remark\3M 

Theorem 19.11 and all results of Sections [7] and [H] had nothing to do with 
optimization (and with the theory of HJB equation of Section [6]). They 
belong to statistics of interacting particles and are of interest in their own 
right. The results hold for all examples given in Section H] (see Theorem 14. ip . 

Recall that we are working with Banach spaces B = Coo(R-^) and D = 
C^(R'^). Combining these results with the theory of HJB equation leads to 
our main result. 
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Theorem 9.2. (i)Suppose {A[t,n,u] : t > 0,/i e Mx,\ e {\i,\2),u E U} 
is a family of linear operators D Coo(R'^) of the form 

A[t, /i, u]f{z) = {h{t, z, ^, u), Vf{z)) + L[t, fi]f{z) (9.7) 

where h G C([0,r] x W, C^2'^^^(R'^ x A^(Ai,A2))) O'lT'd operators L[t, jj] are of 
Levy-Khintchin form i.e. 

L[t, fi]f{z) = hcit, z, /i) V, V)/(^) + {b{t, z, /i), V/(z)) 

(9 8) 

+ / ifiz + y) - fiz) - iVfiz),y)lBMHt,z,fx,dy) 

and satisfy the assumptions in Theorem \8.^ and estimate (13 ■4p for a p E 
(0,2]. 

(a) Let time- dependent Hamiltonian Ht be given by 

Ht{x,p,fi) := max{h(t,x, ^,u)p + J{t,x, fi,u)} (9.9) 

where functions J e C([0,T] x W, C^^'^^^(R'^ x A^(Ai,A2)))- Assume the 
argmax in (19. 9p is unique and it is continuous in t and Lipschitz contin- 
uous in {x,p,fi), uniformly with respect to t,x,fi (fj, in the topology ofD*) 
and bounded p (see e.g. examples at the end of Subsection M . 3\ for the natural 
settings where this condition holds). 

(Hi) Assume 6Ht/6fi{-) exists and is continuous and uniformly bounded. 

(iv) Assume for any {^.} G C^([0,T] x for each X G (Ai,A2), the 
operator curve L[t,fit] '■ D — )■ Coo(R'^) generates a strongly continuous back- 
ward propagator of bounded linear operators U^'^[{fi.}] in the Banach space 
Coo(R'^) with the common invariant domain D, such that, for any [{/i.}], 
and t < s 

max{||f/*'^[{/..}]||o, ||f/''1{/i.}]||c., ||f/''1{/i.}]||ci} < K 

for some constant K > 0, and with their dual propagators f/'''*[{/U.}] preserv- 
ing the set Aix. 

(v) Assume the operators [/^'"[{fi.}] : Coo(R'^) ^ C^(R°'), <t < s, and 

\\U''^[{fim\cu^^)<wis-tm\c^in<^), 

for t < s < T and for all (f) G Coo(R"'), with an integrable positive function 
w on [0, T]. 
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(vi) Assume terminal function V'^ G Coo^(R'^). 
Then, 

(i) for any initial measure /io with a finite moment ofpth order, p G (0, 2], 
there exists a solution {fi.} to equation (11. lip ; 
(a) if the initial conditions of a N players game 



N 



(SxN + ■ ■ ■ + ) 

^ ^1,0 ^N.O' 



converge in D* = (C^(R ))*, as N oo, to a probability law /io G P(R ) 
so that 

n N II ^ ^1 

with a constant ki > and X^q G R'^ converge, as N ^ oo, to a point 
Xi G R'' such that 



1l 

N 



with a constant K2 > 0, the strategy profile u = r{t, x, {fi-}), defined via HJB 
(TTTT^ and U.18\) with {fi.} = {/x.}, is a perfect e-equilibrium in a N players 
game, with 



N 



\J\ 



C([0,T]xW,C^2,2x2(j^dxyK;,j,;,2)) 



+ l|V'^llc^(R'^) + !)• 



Proof, (i) This follows from Theorem 16.31 and Theorem 12.41 

(ii) Let the 1st player choose a different strategy 7, other than F. Denote 
the state dynamics of the 1st player by X^^ or Xj^j, if defined via equation 
(11. 7p with 7 = r or 7 = 7, respectively. Then, 

\V\Q,X^.)[T]-V\Q,X^.)[T.,,^]\ 



< 



E 



Ji(s,x5,/i,[r],r)rfs-E / Ji(s,xiy,,/i,[r_i,7],7)rfs 



N 



+ 



KV'{Xi\)-EV'{X^T) 



By Theorem 19. H 

|v"(o,x,"„)|r|-i/'(o,^i':o)|r-i,' 



- N y 



J\ 



C{[0,T]xU,Cl 



It is clear these estimates hold if we start the game at any time t G [0, T]. □ 
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10 Bibliographical notes 



Below is a very short sketch (far from being complete) of relevant literature 
on LLN for interacting particles, kinetic equations, propagation of chaos, 
mean field and nonlinear Markov control. 

The work on deterministic LLN limits for interacting particles and its 
representation in terms of nonlinear kinetic equations goes back to [65], [T8] . 
The main approach of physicists was via the so-called Bogoliubov or BBGKY 
hierarchies. Similar deduction of kinetic equations for very general system of 
mean- field and kth order interacting systems can be found in [13], [69] and 
[TT] . Quantum analogs of these equations were developed in [12]. The related 
concept of the propagation of chaos was developed by many authors, see e.g. 
[15] , [79] . Especially much work was conducted for interacting diffusions and 
the related McKean-Vlasov nonlinear diffusion processes, see e.g. [71] and 
various recent developments in [ll],[l5], [31], [1] and references therein. 

The method of tagged particle is well established in the theory of inter- 
action, see e.g. [22], [S2], [72], [73], [75] and references therein. However, 
our treatment of this method in conjunction with the theory of infinite- 
dimensional semigroups in a control setting seems to be new. 

The sensitivity analysis for kinetic equations started to be developed re- 
cently, see [HU], [0], [HS], [S2], [SZ], leading both to the rates of convergence 
in numeric algorithms and in theoretical result on the convergence in the 
dynamic LLN and the propagation of chaos. The crucial (especially in the 
context of the present paper) 1/A^-estimates appear in general setting of inter- 
acting particles in [51] for diffusion and stable-like processes with bounded 
coefficients and then was extended to a wide class of interaction with un- 
bounded rates in [52]. Convergence rates are actively studied now, see e.g. 
[25j, [55] and references therein. The algorithmical aspects of particle ap- 
proximations are widely studied from various points of view, see e.g. [22] . 
|47j . [23], [H] and references therein. 

For the introduction to nonlinear Markov evolutions from the physical 
point of view we refer to papers [27] or [HI]. Related aspects of nonlinearity 
in interacting particles are discussed in [BB] and [73]. 

For systematic discussions of the crucial question touched upon in Section 
m on the rigorous link between generators and processes, we refer to books 
[H] and [51] and papers [12], [55], and references therein. Specifically for 
stable like processes see [TD], [H], [SO], [IS] and references therein. For related 
questions see [9]. 
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The seminal papers on mean-field games were cited above many times. 
Some further discussions can be found in video lectures [HI] . As a nice recent 
contribution, let us mention |7], which is devoted to a simple model with 
an underlying process being a standard Brownian motion, where explicit 
solutions and various limiting regimes including long time asymptotic are 
given. Discrete Mean Field Games were studied in [2T], numerical methods 
in pp , |Sn] . Let us stress finally that the concept of the mean field games lies 
in the general domain of mean field and nonlinear Markov control, see e.g. 
[2], [20], [78], Section 11.2 of book [52] and references therein for its quick 
mathematical development, with more application oriented work being [63] . 
[2S|, [IS], [IS], [IZ] (with applications from robot swamps to transportation 
theory and networks). 
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